Rendered at 16:54:36 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
marysol5 5 hours ago [-]
CT logs are funny, not enough people know about them. Whenever I speak to people they're shocked to know that "internal" subdomains get exposed through them.
technion 2 days ago [-]
There's an astounding amount of .DS_Store showing up - I hadn't realised how common it apparently is for people to accidentally upload this.
stingraycharles 2 days ago [-]
It’s a terrible design from Apple to expose this metadata like this, it’s one of my biggest pet peeves.
graemep 1 days ago [-]
KDE does something similar (but not as bad?) with .directory files.
keepamovin 2 days ago [-]
In the early days of the web you could do a search on google like
path:/etc/passwd
Sometimes there were even shadow passwd files with the hashes exposed on the web. Crazy days.
AndReics 3 hours ago [-]
i thought it was still possible!
Luckily security has come a long way, but as shown by the project of OP, we are not quite there yet.
katemaster009 1 days ago [-]
I remember seeing examples like this in security courses.
It was always surprising how many servers accidentally exposed sensitive files.
keepamovin 1 days ago [-]
Yes. Apache misconfigurations were a big one iirc. There were also basic auth files, databases and probably classified/proprietary information.
Similar to how the telephone network used to have all kinds of unsecured entry points that people explored, leading to business phone systems, strange modems, and even international “trunk” lines and operator capabilities.
It was a wild frontier
Elliott-Diy 1 days ago [-]
Is this just re-skinned leakix.net? One of my honeypots for them is showing the same results.
PatchRequest 1 days ago [-]
Nope, i crawl everything myself.
sub-e12 5 hours ago [-]
Awesome tool, fast and right to the point!
sandeepkd 2 days ago [-]
Its interesting and not interesting at the same time based on some of the search results
Almost all of them seem like home projects being deployed with ease in mind than security. The common thread seems to be the fact that most of them are phishing website, not sure if thats a business model to target here?
phoronixrly 2 days ago [-]
So is this the crawler that has been constantly hammering all my applications searching for these files from the very second I first issue a TLS cert for them? Thanks to you I've had to put fail2ban on all my public-facing web servers...
How about you be a good netizen and make it so people can request to be scanned and don't proactively do it, let alone constantly keep hammering them with requests?
bblb 2 days ago [-]
I need to protect against the malicious good guys, the shodans of the world. Peeping inside your windows and trying your front door handles. Querying every string imaginable and port pinging all 65536 of them.
And I need to protect against the actual criminals already inside my house looking for something to steal. Scouting out the easy target to setup their ransomwares.
graemep 1 days ago [-]
Lots of crawlers do this. I have never seen a webserver that does not get a variety of these from obviously different sources. Even just an ssh port will get a lot of malicious login attempts.
phoronixrly 1 days ago [-]
I don't get what you mean by that. Everyone does it so it's OK?
No - scanning without consent is abuse and anyone doing this is malicious regardless of what they tell themselves or anyone else.
graemep 7 hours ago [-]
I am not saying its OK. I am saying it is probably not this crawler that is hammering your site, and definitely not only this crawler. I specifically mentioned malicious attempts at ssh logins.
1 days ago [-]
Avery29 2 days ago [-]
Nice tool. I’d like to understand what kinds of businesses the customers using this website are in.
cvadict 2 days ago [-]
searching for .gov reveals 0 matches... doubt
PatchRequest 1 days ago [-]
I manually blacklisted .gov from being crawled on my side. felt like it wasn't worth the potential trouble
marysol5 5 hours ago [-]
Meanwhile you crawl anybody else?
.gov isn't magically any more able to take you to court than a private org
sandeepkd 2 days ago [-]
My guess is that they ran selective search on the domains which get registered with any registrar, thats the trigger to start the search. .gov domains are not managed by your typical registrar which is selling the domain registration information to all these downstream partners/scavengers (for lack of better word)
jadamson 2 days ago [-]
The OP says it's using CT logs, not new domain registrations. The approach you have in mind would not include subdomains and would be less likely to coincide with a new server being configured.
sandeepkd 2 days ago [-]
Yes CT is explicitly stated source which is why I qualified it with "Guess" for domain registration. There were couple reasons for that -
1. Quite a few websites in the search results where just on HTTP
2. The .gov sites do use public certificate authorities like digicert, verisign, amazon & letencrypt so they would have been captured unless they are removed explicitly
And yes the domain registration would not include subdomains
Luckily security has come a long way, but as shown by the project of OP, we are not quite there yet.
It was always surprising how many servers accidentally exposed sensitive files.
Similar to how the telephone network used to have all kinds of unsecured entry points that people explored, leading to business phone systems, strange modems, and even international “trunk” lines and operator capabilities.
It was a wild frontier
Almost all of them seem like home projects being deployed with ease in mind than security. The common thread seems to be the fact that most of them are phishing website, not sure if thats a business model to target here?
How about you be a good netizen and make it so people can request to be scanned and don't proactively do it, let alone constantly keep hammering them with requests?
And I need to protect against the actual criminals already inside my house looking for something to steal. Scouting out the easy target to setup their ransomwares.
No - scanning without consent is abuse and anyone doing this is malicious regardless of what they tell themselves or anyone else.
.gov isn't magically any more able to take you to court than a private org
1. Quite a few websites in the search results where just on HTTP
2. The .gov sites do use public certificate authorities like digicert, verisign, amazon & letencrypt so they would have been captured unless they are removed explicitly
And yes the domain registration would not include subdomains
manchester.gov.uk uses LE