Commit Graph

152 Commits

Author SHA1 Message Date
Azareal
d0bc74f58f add cocolyze, babbar, surdotly, awesome_bot user agents
reduce matching in the router slightly
use hookgen for router_after_filters
use hookgen for router_pre_route
shorten ua buffer variable name
skip more symbols in initial ua loop
shorten vulnscan string
avoid unneccessary fork in ua logic
2020-05-27 07:08:13 +10:00
Azareal
584a50b461 reuse gzip writers with a sync pool 2020-04-13 14:54:47 +10:00
Azareal
faf215f388 try speeding up static files with brotli
track ref routes for a bit to debug them
hit more bots with MicroNotFound
2020-04-13 13:28:52 +10:00
Azareal
1dc69ed89e add headless chrome ua 2020-04-12 18:33:24 +10:00
Azareal
7d30a66d26 try to filter out bot referrers more effectively
megaindex update didn't save properly
2020-04-09 19:30:51 +10:00
Azareal
49a3837621 add netcraft user agent and alt mj12bot matcher 2020-04-04 19:03:11 +10:00
Azareal
4d45fb8710 add burf.co user agent 2020-04-04 13:21:59 +10:00
Azareal
afb05ab409 add nimbostratus user agent 2020-04-04 13:10:18 +10:00
Azareal
21d09db477 add CCBot user agent 2020-03-31 23:10:29 +10:00
Azareal
fc6544d620 add mattermost ua
save bytes in req dump

phrases:

rename panel_group_promotion_registered_for to panel_group_promotion_reg_for
rename panel_group_promotion_registered_months_suffix to panel_group_promotion_reg_months_suffix
rename panel_group_promotion_registered_days_suffix to panel_group_promotion_reg_days_suffix
rename panel_group_promotion_registered_hours_suffix to panel_group_promotion_reg_hours_suffix
2020-03-28 15:39:05 +10:00
Azareal
b53577a126 fix MicroNotFound
reduce allocs in cdn push
2020-03-27 22:00:29 +10:00
Azareal
3adee57b37 save allocs in router
better logging for bad char in path
router_pre_route hook now returns properly
2020-03-27 16:05:35 +10:00
Azareal
a668cd296b save bytes in the client templates
rename template file and functions to reduce bandwidth usage
replace x-loggedin with theoretically faster x-mem
remove redundant check in ua loop
move extraData initialisation down to where it's needed
2020-03-23 09:14:08 +10:00
Azareal
6b7c51a604 eliminate extraData related allocs in /s/ hot path
shorten generated simple bot bits
2020-03-22 16:24:31 +10:00
Azareal
ade8f82af7 add python and go user agents.
add new bot class for python, go and curl to reduce resource usage.
don't push canonicals for loggedin users
try to avoid pushing google site verify when it isn't necessary
give ahrefs the semrush treatment
2020-03-22 14:37:45 +10:00
Azareal
22ec566a2c add website datenbank user agent
reduce redundancy and fix sublties in StaticFile()
2020-03-21 17:44:33 +10:00
Azareal
6a7b8fd3ee map msnbot to bing and avoid bytes for ahrefs too 2020-03-19 07:21:06 +10:00
Azareal
b66a494f1c Revert "Revert "will a pointer make this faster...?""
This reverts commit 501bcc2425.
2020-03-18 19:21:34 +10:00
Azareal
501bcc2425 Revert "will a pointer make this faster...?"
This reverts commit 8e414486b8.
2020-03-15 21:34:57 +10:00
Azareal
8e414486b8 will a pointer make this faster...? 2020-03-15 17:06:44 +10:00
Azareal
8d19048c0d reduce bytes used on semrush 2020-03-14 12:09:18 +10:00
Azareal
bfea6b740f add cliqz, blexbot, curl user agents
add a /static/ redirect for the slower bots x.x
tweak slack, facebook ua phrases
avoid more allocs for ajax topic
2020-03-13 16:37:18 +10:00
Azareal
aa32bd335c try to avoid as many upfront gzip allocs to avoid wasting resources when they're not needed
avoid gzip for guests on routeAPI
2020-03-12 16:52:26 +10:00
Azareal
64ec820453 add MojeekBot user agent 2020-03-11 18:54:11 +10:00
Azareal
d43bffdec5 experiment with ajax load for topic
track route perf for BadRoute
2020-03-11 13:26:33 +10:00
Azareal
d586220df2 add alternate duckduckgo ua 2020-03-09 20:49:22 +10:00
Azareal
f5c9a34fdd simple top forum topiclist caching
reduce thaw period to 4
fix the _ ua bug
try using a hard period for routeAPI to avoid unneccessary churn for a return string that is unlikely to change
2020-03-08 18:04:37 +10:00
Azareal
6387e2bce0 fix safari ua matcher 2020-03-08 14:36:27 +10:00
Azareal
7d2be466b3 optimise ua tracking
reduce number of bytes used when we don't know the lang
add more spammy domain strings
2020-03-08 14:15:27 +10:00
Azareal
f55a7a23ba add haosou, proximic and mail_ru user agents
add pre_global js hook
try to fix archive.org ua patching
don't word split uas at _
2020-03-08 10:50:23 +10:00
Azareal
d22d6f0b27 add AhrefsBot user agent 2020-03-07 21:08:25 +10:00
Azareal
998cbbfd50 add MJ12bot user agent
add toutiao lang string
2020-03-07 20:46:27 +10:00
Azareal
a80c52861b add toutiao and alternate slack user agents
track route perf for avatars, robots.txt and opensearch
2020-03-07 15:28:25 +10:00
Azareal
46b23ee3a9 log unknown uas 2020-03-07 14:20:42 +10:00
Azareal
b1014d5647 add archive.org user agent 2020-03-07 13:49:44 +10:00
Azareal
ce306d2a4b add slurp, exabot, sogou and alexa user agents
try again to add aspiegelbot
2020-03-07 13:25:50 +10:00
Azareal
24eb9f287c try to add AspiegelBot user agent
tweaked zgrab phrase
2020-03-06 19:12:41 +10:00
Azareal
d2828471cd optimise www redirect check
minor init.js optimisation
avoid gzip for empty obj in alerts endpoint
2020-03-06 13:07:28 +10:00
Azareal
c31ca03a0d use string builder in buildFlexiWhere
add string builder growth hints
track favicon perf
2020-03-05 09:56:45 +10:00
Azareal
46732286e5 optimise route perf tracking
expand route perf tracking to more routes
2020-02-29 11:10:39 +10:00
Azareal
03862d1b09 experiment with more route perf tracking 2020-02-29 09:11:07 +10:00
Azareal
c403b4a85a optimise os tracking
optimise memory tracking

add panel_stats_routes_perf_head phrase
2020-02-28 15:47:43 +10:00
Azareal
ea1037bd63 track favicon stats
experiment with tracking average route performance
temporary error route stub
optimise dumprequest
add DisableAnalytics config setting
fix double hyphens in slugs being mistaken for sql injection
more querygen tests

You wil need to run the updater / patcher for this commit.
2020-02-26 20:34:38 +10:00
Azareal
64335cf1ef optimise lang counter
optimise dumprequest
add more suspicious phrases
capture more router gen errors
2020-02-25 14:25:50 +10:00
Azareal
142359ce11 Add SslSchema config setting.
Reject URL usernames on registration.

Reduce length of char variable name.
2019-11-04 21:55:52 +10:00
Azareal
146e5cff0d Try reducing boilerplate in the generated router. 2019-10-29 17:13:04 +10:00
Azareal
bbef9cead8 Optimise away blocked user queries on profiles for guests.
Shorten some things.
2019-10-20 11:42:19 +10:00
Azareal
4d8c97812d More conversations work.
Add support for AS in columns for SimpleInnerJoin.
Add a referrer policy to improve privacy a little.
Shorten /static/ to /s/ since it comes up so much.
Remove some obsolete code.
Shorten some variable names.
Reduce the amount of boilerplate in the patcher.

Added the RefNoTrack and RefNoRef privacy config settings.

You may need to run the updater / patcher for this commit.
2019-08-14 20:39:04 +10:00
Azareal
ed68c6fc04 Add a simple letter favicon.
Disable Gzip for favicons, it doesn't make them smaller anyway.
Eliminate an allocation for static files.

Fixes #59
2019-07-23 20:34:44 +10:00
Azareal
483ff06775 Try to reduce duplication in the router. 2019-07-12 18:33:25 +10:00