Помогите настроить robots txt

Опубліковано 21 червня, 20179 р

Кто сможет помочь мне настроить правильно robots txt, у меня в поиске более 25000 страниц из них индексируются около 3000 страниц. Все остальное хлам который нужно убирать от поисковиков запретом.

И кто может подсказать чем отличаются robots txt для движка IPB и IPS?

Вот мой robots txt:

User-agent: *
Disallow: /admin/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*%2525
Disallow: /*?s=
Disallow: /*&s=
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Sitemap: https://investmani.ru/sitemap.xml

User-agent: Yandex
Disallow: /admin/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*%2525
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Host: https://investmani.ru
Sitemap: https://investmani.ru/sitemap.xml

User-agent: Google
Disallow: /admin/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*%2525
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Sitemap: https://investmani.ru/sitemap.xml

1

Опубліковано 21 червня, 20179 р

Адміністратор

В IPS

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /datastore/
Disallow: /plugins/
Disallow: /system/
Disallow: /Credits.txt
Disallow: /error.php
Disallow: /upgrading.html
Disallow: /login/
Disallow: /register/
Disallow: /lostpassword/
Disallow: /online/
Disallow: /contact/
Disallow: /support/
Disallow: /profile/
Disallow: /?tab=*
Disallow: /index.php?*
Disallow: /*?app=*
Disallow: /*sortby=*
Disallow: /*/?do=download
Disallow: /profile/*/?do=*
Sitemap: https://ipb-board.com/sitemap.php

User-agent: Yandex
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /datastore/
Disallow: /plugins/
Disallow: /system/
Disallow: /Credits.txt
Disallow: /error.php
Disallow: /upgrading.html
Disallow: /login/
Disallow: /register/
Disallow: /lostpassword/
Disallow: /online/
Disallow: /contact/
Disallow: /support/
Disallow: /profile/
Disallow: /?tab=*
Disallow: /index.php?*
Disallow: /*?app=*
Disallow: /*sortby=*
Disallow: /*/?do=download
Disallow: /profile/*/?do=*
Host: https://ipb-board.com
Sitemap: https://ipb-board.com/sitemap.php

1

Опубліковано 22 червня, 20179 р

Автор

А если я в IPB такой robots.txt установлю работать будет?

Опубліковано 22 червня, 20179 р

Адміністратор

10 часов назад, sergstesh сказал:

А если я в IPB такой robots.txt установлю работать будет?

Нажаль нет.

Вот став на IPB

User-agent: *
Disallow: /xSmaREtinkA/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*%2525
Disallow: /*?s=
Disallow: /*&s=
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Sitemap: http://ipb-board.com/sitemap.xml

User-agent: Yandex
Disallow: /xSmaREtinkA/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*%2525
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Host: ipb-board.com
Sitemap: http://ipb-board.com/sitemap.xml

User-agent: Google
Disallow: /xSmaREtinkA/
Disallow: /cache/
Disallow: /converge_local/
Disallow: /hooks/
Disallow: /ips_kernel/
Disallow: /retail/
Disallow: /public/style_captcha/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras&section=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=core&module=search&do=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*app=forums&module=extras&section=stats&do=who
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*view__getnewpost$
Disallow: /*view__getlastpost$
Disallow: /*view__old$
Disallow: /*view__new$
Disallow: /*view__getfirst$
Disallow: /*view__getprevious$
Disallow: /*view__getnext$
Disallow: /*view__getlast$
Disallow: /*&view=getnewpost$
Disallow: /*&view=getlastpost$
Disallow: /*&view=old$
Disallow: /*&view=new$
Disallow: /*&view=getfirst$
Disallow: /*&view=getprevious$
Disallow: /*&view=getnext$
Disallow: /*&view=getlast$
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*%2525
Disallow: /user/
Disallow: /members/
Disallow: /index.php?app=members
Disallow: /index.php?app=core&module=global&section=login&do=deleteCookies
Disallow: /index.php?app=forums&module=extras&section=rating
Disallow: /index.php?app=forums&module=forums&section=markasread
Disallow: /index.php?app=core&module=search&do=search&search_tags=
Sitemap: http://ipb-board.com/sitemap.xml

1

Опубліковано 22 червня, 20179 р

Автор

Точно такой как у меня и стоит, но почему-то роботы обнаруживают не нужные страницы и удаляют их из поиска. А вот такие к примеру индексируются:

https://investmani.ru/index.php?app=core&module=search&do=user_activity&mid=5&userMode=title&search_app=forums&search_app_filters[forums][searchInKey]=&search_app_filters[forums][sortKey]=date&search_app_filters[forums][sortDir]=1

https://investmani.ru/index.php?app=core&module=search&do=user_activity&mid=231&search_app=forums&userMode=content&search_app_filters[forums][searchInKey]=&search_app_filters[forums][sortKey]=title&search_app_filters[forums][sortDir]=&st=50

https://investmani.ru/index.php?app=core&module=search&do=user_activity&mid=52&search_app=forums&userMode=content&search_app_filters[forums][searchInKey]=&search_app_filters[forums][sortKey]=date&search_app_filters[forums][sortDir]=1&st=75

Может robots.txt (указанный выше) не все запреты в себе несет, или это не из-за него?

Опубліковано 10 липня, 20179 р

Вот здесь можно скачать уже настроенный файл robots.txt:

Помогите настроить robots txt

Рекомендовані відповіді

Акаунт

Навігація

Пошук

Налаштування push-сповіщень браузера

Chrome (Android)

Chrome (Комп'ютер)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Комп'ютер)

Firefox (Android)

Firefox (ПК)