mirrors/farside
1
0
Fork 0
mirror of https://github.com/benbusby/farside.git synced 2025-04-25 13:10:02 +00:00
Code Wiki Activity

Filter against cloudflare proxy (not nameservers)

Browse source 
The previous approach to filtering cloudflare instances was checking if
cloudflare nameservers were assigned to that domain. I believe this was
too harsh of a filter, as quite a few instances were removed even though
they were not using cloudflare's proxying feature.

To solve this, the filtering script has been updated to manually fetch
each IP associated with a domain and see if it returns the 1003 error
that cloudflare proxied sites return when queried directly.
This commit is contained in:
Ben Busby 2022-12-09 10:57:39 -07:00
parent 47852145e3
commit 1c83d822d6
No known key found for this signature in database
GPG key ID: B9B7231E01D924A1
2 changed files with 176 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

33
tools/un-cloudflare.sh
View file

@ -7,12 +7,27 @@ file="services-full.json"
while read -r line; do
if [[ "$line" == "\"https://"* ]]; then
domain=$(echo "$line" | sed -e "s/^\"https:\/\///" -e "s/\",//" -e "s/\"//")
ns=$(dig ns "$domain" || true)
if [[ "$ns" == *"cloudflare"* ]]; then
echo "\"$domain\" using cloudflare, skipping..."
elif [[ "$ns" != *"NOERROR"* ]]; then
echo "Unable to verify records for \"$domain\", skipping..."
else
ips=$(dig "$domain" +short || true)
cf=0
echo "$domain"
for ip in $ips
do
echo " - $ip"
resp=$(curl -s "$ip")
# Cloudflare does not allow accessing sites using their IP,
# and returns a 1003 error code when attempting to do so. This
# allows us to check for sites using Cloudflare for proxying,
# rather than just their nameservers.
if [[ "$resp" == *"error code: 1003"* ]]; then
cf=1
echo " ! Using cloudflare proxy, skipping..."
break
fi
done
if [ $cf -eq 0 ]; then
echo "$line" >> out.json
fi
else
@ -21,7 +36,7 @@ while read -r line; do
done <$file
# Remove any trailing commas from new instance lists
sed -i -e ':begin' -e '$!N' -e 's/,\n]/\n]/g' -e 'tbegin' -e 'P' -e 'D' out.json
#sed -i -e ':begin' -e '$!N' -e 's/,\n]/\n]/g' -e 'tbegin' -e 'P' -e 'D' out.json
cat out.json | jq --indent 2 . > services.json
rm -f out.json
#cat out.json | jq --indent 2 . > services.json
#rm -f out.json