vacplusplus / serposcope Goto Github PK

I formerly had a bing module I found on another site and added it to my uniz/apache instalation of version 1. Was wondering if any of you geniuses have written a bing module for version 2 for windows?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/48

Translated into Japanese.

This issue has been cloned from: serphacker#222

solve #196
When using proxies in docker container, feature to using hostname for http proxy will be needed.

Ipv4 validation seems to be effective only for BIND ips configuration.
So I moved it.

How about this?

This issue has been cloned from: serphacker#198

That would be nice, instead of deleting a group, to be able to disable it, so that we can keep its history, without sending requests (thus, not be blocked)

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/75

I faced a problem that the SERPOSCOPE_DB_URL environmental variable does not work.

The replace_param() function in docker/entrypoint.sh has an issue which does not handles special character & correctly.

Example:

jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASS&allowMultiQueries=true
will be converted to
jdbc:mysql://HOSTNAME/DATABASE?user=USERpassword=PASSallowMultiQueries=true

So, I fixed the issue by adding piped sed command which perform escaping & characters.

This issue has been cloned from: serphacker#179

It would be useful to add a small note on each keyword, specifically the Monthly number of searches, so we can have better insights on each keyword's importance directly from Serposcope.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/106

About a week ago, mobile terms that were working fine all the time now report no data at all

Any idea?

Since I don't see anything about the CVE, I'm going to open an issue.

According to a quick search, Log4j is used by Serposcope scraper: https://github.com/serphacker/serposcope/blob/master/scraper/pom.xml#L39-L54
Can anyone confirm this? I don't have enough knowledge in Java...

What is the plan to fix it?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/246

https://forum.serphacker.com/post.php?tid=169

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/123

Before each run, check for valid proxies from URL of proxy list

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/55

First of all I want to say thanks to all creators, you did a great job. Especially thanks for dockerizing!
We have reviewed your system with our team and we would like to contribute to development process.

Here is a request:
Lets say I have 100k pages (unique urls) with 20 keywords per each page. So I will create 100k groups of keywords to be able to track position of every specific url. But it is going to be a very big mess with 100k groups and viewing statistic. So it would be nice to organize groups by some kind of sub-groups.

Suggested solution: Tag cloud!
Probably the best way of solving this problem is to have ability to create and add tags for every group (or maybe for every keyword). With this kind of system you can create any structure you want for groups of keywords and then view statistic by searching by tags.

What do you think about this feature?
Do you have any other solutions?
What do you think is the best way to implement it?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/94

I have set my time for dealy for google from 15 to 60s. Still program fetches like few entries right from start. Not sure if delay works. Not using any proxies.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/204

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/44

Provide a Mac OSX package, .pkg or .dmg ?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/45

Stop tracking keywords of a specific group but keep it viewable.

https://forum.serphacker.com/viewtopic.php?pid=1036

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/154

Add default search options at group level, this way all new searches will inherit the group options instead of global options if defined.

Also page number and number of search result should be moved from google network/scrap options to search options level so we can create different searches with various page/result depth.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/52

Feature request: schedule each group independently via cron.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/49

Is there any way to type the captcha by hand? It would be very useful for those of us who check the ranks occasionally or when captcha-solving servers are down.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/105

Right now Serposcope assigns these points based on SERP position:

rank #1 = 100 points
rank #2 = 90 points
rank #3 = 80 points
rank #4-#5 = 70 points
rank #6-#10 = 60 points
rank #11-#20 = 40 points
rank #21-#30 = 30 points
rank #31-#50 = 20 points
rank #51-#100 = 10 points
rank #100+ = 5 points
unranked = 0 points

This is useful although I'd suggest ranking more like SERP Click Through Rate studies show:

So it would look more like:

rank #1 = 100 points
rank #2 = 70 points
rank #3 = 60 points
rank #4-#5 = 50 points
rank #6-#10 = 40 points
rank #11-#20 = 22 points
rank #21-#30 = 15 points
rank #31-#50 = 10 points
rank #51-#100 = 5 points
rank #100+ = 1 points
unranked = 0 points

(You can play around with it here.)

@Roman-121 suggested also taking into account the Search Volume in Rankings which makes a lot of sense although it's a bit more complex to develop yet it would fulfill this suggestion too.

In any case, it would be a good idea to be able to set your own ranking through serposcope.conf, that should be easy enough. I'd do it myself but know nothing about Java.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/121

It would be nice if there could be a way to sent out automatic reports (in .pdf) of the keywords position to an email address. For example per group or keyword. In that way i can send out statistics of the keyword position and competitor position to my peer or customer.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/163

I extended the message files and replaced some hard coded text

This issue has been cloned from: serphacker#162

The docs are Docker docs are incorrect, the example SERPOSCOPE_DB_URL provided doesn't work. Ampersands need to be escaped for sed to function correctly in entrypoint.sh.

This issue has been cloned from: serphacker#221

When adding a new website to a group, it seems that there is only support for .com. Is there an ability to add the feature for .co and others.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/205

Allow CRON to be run every X days/weekly so tasks can last more than 24 hours

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/51

Does anyone actually have the pause/delay working? Despite having settings, the crawler just pummels google with no delay.

...of course, this is creating captures and everything fails

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/262

A way to have separate option to monitor what happens in the paid advertising Page 1 only.

This is good for identifying what keywords have high paying ads and monitor fluctuation of advertisers.
If someone wants to build say adsense sites this option/feature will help them to see what keywords are good for building adsense sites, the more ads the better. And for those who want to use adwords paid advertising can use the keywords that have less or zero competition.

So, if you consider this feature good to implement, I'd like to have for keywords a separate panel where I can see for each keyword the total number of advertisers per specific day
Example:
keyword 1 >> 5 ads
keyword 2 >> 0 ads
keyword 3 > 7 ads
...etc...

A simple monitor for number of ads is fine. Basically let us create group, import bulk keywords and have info for all keywords in same panel. A filter keyword based on variables (say show only keywords with > 2 ads) option is preferred to have but if it's not possible a export to csv is enough I think. Also, a cron for 1 week check is enough I think, but only if we can setup separate cron for each group.

Let me know if possible.
Thanks

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/61

Hi

what would be great is to have a picto to tell if result is a local/geo/maps result or a natural/serp result.

And best would be to have, if it's the case the local result position AND the natural result position for a keyword.

regards

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/99

Hello

The UI may be readable on a smartphone.
Some possible and rapid improvements for better adaptative UI (and not responsive, which would need a refactoring) :

• adaptative tables (remove all black areas)
• buttons can be pushed : launch manual crawl button at least, clean bad proxy button, better column sorting (I zoom to be able to click on the the column header)

regards

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/192

Guys, what do you think about implementing API to import keywords?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/93

Hello,

I'm using serposcope in docker, so my proxies are linked to my container, one by ip, but, I didn't get the ip of the proxies .

I've only the hostname .

So, I've tryied to add this proxies :
http#proxy1#3128 http#proxy2#3128

But I got the error :
IPv4 invalide ligne http#proxy1#3128

So, I thing the proxy add feature is not compatible with hostnames :/ .

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/196

Great software !

Is there any possibility to have a command line to start a cron job, quitting at the end ?
This allows to set cron using system scheduled task. Might be helpful in some config where the webserver has side effect (seems synology NAS would not hibernate anymore).
It might enable few other use cases.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/208

This issue has been cloned from: serphacker#209

Hi,
I'm new to this application and wonder is it possible to select or deselect keywords when you run for a search? I noticed that all keywords are searched all the time no matter if they are ticked on or off.
Is it also possible to exclude websites from search?
Thanks! David

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/195

Hey there...

One feature that would be absolutely phenomenal would be the ability to export the full SERP results for a given project / time period. This is a feature that I have only found on one product after testing many... and that product is now dead (AWR Enterprise Desktop). AWR Cloud can still do it... but its much slower and costly.

Specific requirement eg:

1. Run 'x' number of keywords and pull down the top 'x' rankings (usually top 20 for us)

2. When complete, be able to export all of the raw ranking data via CSV. So keyword 'x' would have '20' rows if I were pulling down the top 20 rankings.

3. Why? Our clients absolutely love the ability to see the market as a whole to see not only how they rank for a given (large) keyword set... but also what others are doing in the market as well.

4. We pivot and leverage the data in many different ways to be able to show all sorts of different rank lenses... but the critical need is having the raw ranking data exported or stored in an accessible way... which serposcope is so close to having now it seems. The data is there.. its just the ability to export the full set.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/210

This issue has been cloned from: serphacker#194

On each scrape also extract data like:
Page Title
Moz: PA, DA
Alexa stats per page and per domain
Websearch Index (total amount of results for a Google search for 'keyword'.)
SEMRush Domain Reports
Social Stats: Gogle+ FB, Twitter, etc
here is a powerful open source PHP library that can help you out: https://github.com/eyecatchup/SEOstats

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/67

MAKE BACKUP option for the database and IMPORT because it dies too often and somehow deletes db

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/248

On the website's table view the shortest date shortcut is for '30 days' and on some devices like mobile (or 11" macbook) the 30 days table is way too wide that force people to scroll to see the latest result.

Somehow the need to scroll is a little bit annoying, especially when you just want to take a quick look. (Yes, you can set a custom range, but it takes a lot of click, and people are lazy)

There is 2 ways solve this:

• Add smaller range as a shortcut
• Reverse the website's table horizontally, so the latest data shown on the leftest

Thank you for your time and attention, Serposcope is really useful!

This issue has been cloned from: serphacker#165

We got featured snippet for few queries, but can't see ranking on Serposcope.
IF we get featuresd snippets, is there any settings that needs to be change?
or is it impossible to know the ranking when we get featured snippets?

Example
We got featured snippet for "ダイレクトメール" in June 18. *pic1
This query also show up 1 in google, however Serposcope shows "out" after we got featured snippets. *pic2

Since we rely on Serposcope, we would be grateful if you could advise us what we need to do.
Our domain - https://media-radar.jp/

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/261

Is it possible to develop more features, start a new task and test just one keyword?

Thanks

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/213

ipv6 http proxy always returns an error
ipv4 proxy - ok

serposcope support ipv6?

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/199

Example:

GET https://www.google.com/search?q=rachat+d%27or&uule=w+CAIQICIfbGlsbGUsbm9yZC1wYXMtZGUtY2FsYWlzLGZyYW5jZQ==&start=10 on datacenter X.X.X.X via proxy:direct try 1

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/56

We are currently facing a major issue which we were able to find only after a number of manual trials.

Example
On mobile, we get the featured snippets for search queries
"Toronto trade fairs"
"Textile trade shows in Malaysia"

Our domain - https://10times.com

We have deployed the same settings in Serposcope - Mobile | Global

From the past 1 month, we see the "out" only for the keywords where we have featured snippets on mobile.

If we have 2 placements, one in the featured snippet and other in list, ranking from the list is considered by Serposcope.

Since we actively rely on Serposcope, could you advise us if there is any setting that needs to be changed to take featured snippet into consideration.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/223

This is a request for SERP Alert feature.
Considering that an install can be with lots of projects very easy, a SERP Alert feature is helpful.

Nothing complicated, just a way to add a notification in main dashboard and/or send email notifications to users that created the Group when URL increases/decreases position in serp significantly.

As an example, send alert notification if a webpage rankings:

• jumps up say 10 positions in serps in top 20 only ( say from position 20 to position 10) = May be good sign of google update on that keyword
• jumps down 15 positions in serps in top 20 only ( say from position 7 to position 22) = Sign of deindexation on specific keyword
  All above to have setup per each group

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/62

When using multiple threads, runDB.update* is called multiple times by multiple threads and may stress the database uselessly. Better start a new dedicated threads which call update() every 10 seconds.

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/84

Hello, this app work fine, thanks for it. I have question - can i use this app as only API?
My idea is send request via another script (like php) -
http://127.0.0.1:7134?keyword=xxxxxxx

And i need get back (great idea is json) back first 20 - 30 results with position...
1 -> http:////
2 -> http:////

And i need using proxy db... Is that real? Thanks

This issue has been cloned from: https://github.com/serphacker/serposcope/issues/206