7 Notes, 0 Own notes, 0 Own private notes, 0 Own public notes
Tag: #chatgpt
1 matching note
Notes
BigFoot McFly BigFoot McFly #blog #chatgpt

ChatGPT is a busy bee..

Wow, in less then 12 hours, after i created the mirror repository and uploaded my side project to github, chatgpt did crawl my live site, which was only linked in the github repository :D

Well, it was linked in the main repository as well, but that has been up for days, so it monitors github pretty intensely...

It was not denied in the robots.txt, i just wonder, if it would have been banned, would it have respected the ban?

Some notes:

  • It did first read the robots.txt

  • After that the crawler cam from a different IP

  • It did download all html, javascript, images, css even fonts, basically what it could find

  • It did not try to fill any form, only did GET requests

access log ... 
 14.227.36.37 - - [20/Oct/2025:22:27:46 +0000] "GET /robots.txt HTTP/1.1" 200 55 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +h
 2ttps://openai.com/searchbot"
 320.171.207.65 - - [20/Oct/2025:22:27:47 +0000] "GET / HTTP/1.1" 200 35202 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 420.171.207.65 - - [20/Oct/2025:22:27:49 +0000] "GET /notes/bigfoot-mcfly/prevent-page-expired-errors-in-laravel-pest-testing-with-actingas HTTP/1.1" 200 29763 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 520.171.207.65 - - [20/Oct/2025:22:27:53 +0000] "GET /notes/bigfoot-mcfly/always-use-chattr-i-on-backup-files HTTP/1.1" 200 28572 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 620.171.207.65 - - [20/Oct/2025:22:27:55 +0000] "GET /notes/benedek-gerzson/how-does-laravel-withoutmodelevents-work HTTP/1.1" 200 29364 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 720.171.207.65 - - [20/Oct/2025:22:27:58 +0000] "GET /storage/avatars/01K7PGM49GYJYKJ9E16AZGTJ6J.png HTTP/1.1" 200 231773 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 820.171.207.65 - - [20/Oct/2025:22:27:59 +0000] "GET /notes/bigfoot-mcfly/filamentphp-panel-with-empty-path-name-and-a-custom-front-page HTTP/1.1" 200 29410 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
 920.171.207.65 - - [20/Oct/2025:22:28:01 +0000] "GET /js/filament/support/support.js?v=4.1.7.0 HTTP/1.1" 200 55524 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1020.171.207.65 - - [20/Oct/2025:22:28:03 +0000] "GET /notes/bigfoot-mcfly/laravel-livewire-upload-error-behind-nginxcloudflare HTTP/1.1" 200 29992 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1120.171.207.65 - - [20/Oct/2025:22:28:05 +0000] "GET /storage/avatars/01K6W7KCNQBW9BAM2ZGVBK584E.jpg HTTP/1.1" 404 2318 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1220.171.207.65 - - [20/Oct/2025:22:28:06 +0000] "GET /livewire/upload-file?expires=1760438926&signature=26dd9c5dd9156ea312bf70917931d1c2dc3ea1cfead611bf5523baa7ba706a25%E2%80%9D HTTP/1.1" 405 1023 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1320.171.207.65 - - [20/Oct/2025:22:28:07 +0000] "GET /notes/bigfoot-mcfly/laravel-model-local-scoping-with-attributes HTTP/1.1" 200 29771 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1420.171.207.65 - - [20/Oct/2025:22:28:08 +0000] "GET /login HTTP/1.1" 200 6285 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1520.171.207.65 - - [20/Oct/2025:22:28:10 +0000] "GET /vendor/livewire/livewire.min.js?id=df3a17f2 HTTP/1.1" 200 58572 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1620.171.207.65 - - [20/Oct/2025:22:28:11 +0000] "GET /build/assets/app-BmsqrX6M.js HTTP/1.1" 200 1 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1720.171.207.65 - - [20/Oct/2025:22:28:12 +0000] "GET /build/assets/scriptum-CSnYTD7J.js HTTP/1.1" 200 168 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1820.171.207.65 - - [20/Oct/2025:22:28:13 +0000] "GET /users/bigfoot-mcfly HTTP/1.1" 200 34443 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
1920.171.207.65 - - [20/Oct/2025:22:28:14 +0000] "GET /js/filament/actions/actions.js?v=4.1.7.0 HTTP/1.1" 200 439 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2020.171.207.65 - - [20/Oct/2025:22:28:15 +0000] "GET /livewire/upload-file?...%27 HTTP/1.1" 405 1023 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2120.171.207.65 - - [20/Oct/2025:22:28:17 +0000] "GET /js/filament/schemas/schemas.js?v=4.1.7.0 HTTP/1.1" 200 1279 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2220.171.207.65 - - [20/Oct/2025:22:28:17 +0000] "GET /register HTTP/1.1" 200 6821 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2320.171.207.65 - - [20/Oct/2025:22:28:19 +0000] "GET /users/benedek-gerzson HTTP/1.1" 200 29824 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2420.171.207.65 - - [20/Oct/2025:22:28:20 +0000] "GET /js/filament/tables/tables.js?v=4.1.7.0 HTTP/1.1" 200 2492 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2520.171.207.65 - - [20/Oct/2025:22:28:21 +0000] "GET /js/filament/filament/app.js?v=4.1.7.0 HTTP/1.1" 200 4450 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2620.171.207.65 - - [20/Oct/2025:22:28:22 +0000] "GET /storage/avatars/_guest.svg HTTP/1.1" 200 5781 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2720.171.207.65 - - [20/Oct/2025:22:28:24 +0000] "GET /js/filament/filament/echo.js?v=4.1.7.0 HTTP/1.1" 200 29165 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2820.171.207.65 - - [20/Oct/2025:22:28:25 +0000] "GET /js/filament/notifications/notifications.js?v=4.1.7.0 HTTP/1.1" 200 2764 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2920.171.207.65 - - [20/Oct/2025:22:28:26 +0000] "GET /build/assets/theme-C-fBfsLR.css HTTP/1.1" 200 80042 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3020.171.207.65 - - [20/Oct/2025:22:28:27 +0000] "GET /build/assets/app-DYP3tEh-.css HTTP/1.1" 200 35485 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3120.171.207.65 - - [20/Oct/2025:22:28:28 +0000] "GET /fonts/filament/filament/inter/index.css?v=4.1.7.0 HTTP/1.1" 200 633 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3220.171.207.65 - - [20/Oct/2025:22:28:29 +0000] "GET /vendor/livewire/livewire.min.js.map HTTP/1.1" 200 584196 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3320.171.207.65 - - [20/Oct/2025:22:28:31 +0000] "GET /fonts/filament/filament/inter/inter-greek-wght-normal-IRE366VL.woff2 HTTP/1.1" 200 18996 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3420.171.207.65 - - [20/Oct/2025:22:28:32 +0000] "GET /fonts/filament/filament/inter/inter-latin-wght-normal-NRMW37G5.woff2 HTTP/1.1" 200 48256 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3520.171.207.65 - - [20/Oct/2025:22:28:33 +0000] "GET /fonts/filament/filament/inter/inter-vietnamese-wght-normal-CE5GGD3W.woff2 HTTP/1.1" 200 10252 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3620.171.207.65 - - [20/Oct/2025:22:28:34 +0000] "GET /fonts/filament/filament/inter/inter-latin-ext-wght-normal-HA22NDSG.woff2 HTTP/1.1" 200 85068 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3720.171.207.65 - - [20/Oct/2025:22:28:36 +0000] "GET /fonts/filament/filament/inter/inter-greek-ext-wght-normal-EOVOK2B5.woff2 HTTP/1.1" 200 11232 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3820.171.207.65 - - [20/Oct/2025:22:28:37 +0000] "GET /fonts/filament/filament/inter/inter-cyrillic-ext-wght-normal-IYF56FF6.woff2 HTTP/1.1" 200 25960 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
3920.171.207.65 - - [20/Oct/2025:22:28:38 +0000] "GET /fonts/filament/filament/inter/inter-cyrillic-wght-normal-JEOLYBOO.woff2 HTTP/1.1" 200 18748 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
4020.171.207.65 - - [21/Oct/2025:01:17:02 +0000] "GET /favicon.ico HTTP/1.1" 200 8679 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"
2025-10-21 08:26:01
permalink
Powered by: Scriptum Version: v0.7.0-dev