b9d94d0888
hoist runnables shared among all routes in a route group to the route group to reduce code duplication
...
rewrite router gen to make indentation maintenance easier
2021-03-18 10:22:28 +10:00
75561508c8
don't log referer when DNT header is set
...
skip Cookie headers in req logs
2021-03-02 18:47:36 +10:00
4126e8ed0c
add qwant, xenforo, twingly, linkfluence, and new toutiao user agent.
...
add Users.CountSearch test cases.
add Users.SearchOffset test cases.
add more Users.Count test cases.
add more Users.BulkGetByName test cases.
reduce boilerplate in other tests.
fix bugs in Users.CountSearch and Users.BulkGetByName
2021-03-02 17:22:32 +10:00
e49a79014b
Log suspicious requests in a seperate log file.
...
Optimise the request logger.
Avoid converting from byte slice to string in router gen.
2021-02-27 16:52:51 +10:00
de2221f243
fix main.css not being served on /topics/
...
panic when we try to reference an agent which doesn't exist in gen router init
2021-02-18 09:37:20 +10:00
1f0713bb27
add wappalyzer, yacy user agents
...
fix mj12bot user agent parsing
numbers should now work in user agent marks
2021-02-17 15:27:56 +10:00
0740b3bf14
Reduce user agent mark boilerplate in router gen.
...
Fix PetalBot detection.
2021-02-17 10:07:11 +10:00
74e0ce492d
Weekly Views.
...
Add avif mime extension.
Optimise skipping watched files which haven't been changed.
Block zgrab to save resources.
Reduce text and boilerplate.
Add route_topic_list_weekviews_start hook.
Add topic_list.week_views_filter phrase.
Changed footer_powered_by phrase.
You will need to run the updater / patcher for this commit.
2021-01-03 07:46:24 +10:00
75d1d6fcbe
add telegram, domcop, maui and alt aspiegel user agents
...
reduce boilerplate in routergen
2020-05-29 08:41:19 +10:00
d0bc74f58f
add cocolyze, babbar, surdotly, awesome_bot user agents
...
reduce matching in the router slightly
use hookgen for router_after_filters
use hookgen for router_pre_route
shorten ua buffer variable name
skip more symbols in initial ua loop
shorten vulnscan string
avoid unneccessary fork in ua logic
2020-05-27 07:08:13 +10:00
584a50b461
reuse gzip writers with a sync pool
2020-04-13 14:54:47 +10:00
faf215f388
try speeding up static files with brotli
...
track ref routes for a bit to debug them
hit more bots with MicroNotFound
2020-04-13 13:28:52 +10:00
1dc69ed89e
add headless chrome ua
2020-04-12 18:33:24 +10:00
7d30a66d26
try to filter out bot referrers more effectively
...
megaindex update didn't save properly
2020-04-09 19:30:51 +10:00
49a3837621
add netcraft user agent and alt mj12bot matcher
2020-04-04 19:03:11 +10:00
4d45fb8710
add burf.co user agent
2020-04-04 13:21:59 +10:00
afb05ab409
add nimbostratus user agent
2020-04-04 13:10:18 +10:00
21d09db477
add CCBot user agent
2020-03-31 23:10:29 +10:00
fc6544d620
add mattermost ua
...
save bytes in req dump
phrases:
rename panel_group_promotion_registered_for to panel_group_promotion_reg_for
rename panel_group_promotion_registered_months_suffix to panel_group_promotion_reg_months_suffix
rename panel_group_promotion_registered_days_suffix to panel_group_promotion_reg_days_suffix
rename panel_group_promotion_registered_hours_suffix to panel_group_promotion_reg_hours_suffix
2020-03-28 15:39:05 +10:00
b53577a126
fix MicroNotFound
...
reduce allocs in cdn push
2020-03-27 22:00:29 +10:00
3adee57b37
save allocs in router
...
better logging for bad char in path
router_pre_route hook now returns properly
2020-03-27 16:05:35 +10:00
a668cd296b
save bytes in the client templates
...
rename template file and functions to reduce bandwidth usage
replace x-loggedin with theoretically faster x-mem
remove redundant check in ua loop
move extraData initialisation down to where it's needed
2020-03-23 09:14:08 +10:00
6b7c51a604
eliminate extraData related allocs in /s/ hot path
...
shorten generated simple bot bits
2020-03-22 16:24:31 +10:00
ade8f82af7
add python and go user agents.
...
add new bot class for python, go and curl to reduce resource usage.
don't push canonicals for loggedin users
try to avoid pushing google site verify when it isn't necessary
give ahrefs the semrush treatment
2020-03-22 14:37:45 +10:00
22ec566a2c
add website datenbank user agent
...
reduce redundancy and fix sublties in StaticFile()
2020-03-21 17:44:33 +10:00
6a7b8fd3ee
map msnbot to bing and avoid bytes for ahrefs too
2020-03-19 07:21:06 +10:00
b66a494f1c
Revert "Revert "will a pointer make this faster...?""
...
This reverts commit 501bcc2425
2020-03-18 19:21:34 +10:00
501bcc2425
Revert "will a pointer make this faster...?"
...
This reverts commit 8e414486b8
2020-03-15 21:34:57 +10:00
8e414486b8
will a pointer make this faster...?
2020-03-15 17:06:44 +10:00
8d19048c0d
reduce bytes used on semrush
2020-03-14 12:09:18 +10:00
bfea6b740f
add cliqz, blexbot, curl user agents
...
add a /static/ redirect for the slower bots x.x
tweak slack, facebook ua phrases
avoid more allocs for ajax topic
2020-03-13 16:37:18 +10:00
aa32bd335c
try to avoid as many upfront gzip allocs to avoid wasting resources when they're not needed
...
avoid gzip for guests on routeAPI
2020-03-12 16:52:26 +10:00
64ec820453
add MojeekBot user agent
2020-03-11 18:54:11 +10:00
d43bffdec5
experiment with ajax load for topic
...
track route perf for BadRoute
2020-03-11 13:26:33 +10:00
d586220df2
add alternate duckduckgo ua
2020-03-09 20:49:22 +10:00
28f210371e
try to optimise routeAPI
2020-03-09 15:16:44 +10:00
f5c9a34fdd
simple top forum topiclist caching
...
reduce thaw period to 4
fix the _ ua bug
try using a hard period for routeAPI to avoid unneccessary churn for a return string that is unlikely to change
2020-03-08 18:04:37 +10:00
6387e2bce0
fix safari ua matcher
2020-03-08 14:36:27 +10:00
7d2be466b3
optimise ua tracking
...
reduce number of bytes used when we don't know the lang
add more spammy domain strings
2020-03-08 14:15:27 +10:00
f55a7a23ba
add haosou, proximic and mail_ru user agents
...
add pre_global js hook
try to fix archive.org ua patching
don't word split uas at _
2020-03-08 10:50:23 +10:00
d22d6f0b27
add AhrefsBot user agent
2020-03-07 21:08:25 +10:00
998cbbfd50
add MJ12bot user agent
...
add toutiao lang string
2020-03-07 20:46:27 +10:00
a80c52861b
add toutiao and alternate slack user agents
...
track route perf for avatars, robots.txt and opensearch
2020-03-07 15:28:25 +10:00
46b23ee3a9
log unknown uas
2020-03-07 14:20:42 +10:00
b1014d5647
add archive.org user agent
2020-03-07 13:49:44 +10:00
ce306d2a4b
add slurp, exabot, sogou and alexa user agents
...
try again to add aspiegelbot
2020-03-07 13:25:50 +10:00
24eb9f287c
try to add AspiegelBot user agent
...
tweaked zgrab phrase
2020-03-06 19:12:41 +10:00
d2828471cd
optimise www redirect check
...
minor init.js optimisation
avoid gzip for empty obj in alerts endpoint
2020-03-06 13:07:28 +10:00
c31ca03a0d
use string builder in buildFlexiWhere
...
add string builder growth hints
track favicon perf
2020-03-05 09:56:45 +10:00
46732286e5
optimise route perf tracking
...
expand route perf tracking to more routes
2020-02-29 11:10:39 +10:00
Azareal