How to configure data source DEPATISnet about patzilla

[amotl]

Dear @ElvezPelvez,

thanks for writing in.

Introduction

PatZilla will always require access to OPS as it is the single data source available for displaying bibliographic data.

However, PatZilla should be able to offer searching at DEPATISnet in its default configuration

datasources = ops, depatisnet

so let's check what might be wrong with your setup.

Investigation

May I ask you whether you see this in the data source chooser widget in the user interface?

After switching to "DPMA" there, you should be able to query DEPATISnet through the corresponding data source adapter depatisnet.py.

Outlook

To be able to display bibliographic data from DEPATISnet in order to skip OPS on that, this data source adapter would have to be extended to actually acquire the bibliographic details. Currently, it just submits the query expression, takes the list of document numbers from the response and passes on the torch to the display subsystem which is currently solely based on OPS.

While extending depatisnet.py is definitively doable up to a certain degree, it's currently not on our list of priorities. Saying that, we might come back to this in the future, actually depending on pressure vs. opportunities ;].

Advice

Regarding access to OPS, we would recommend getting an account there for provisioning the PatZilla configuration with the corresponding authentication credentials. Regarding searching at DEPATISnet, we are confident to be able to resolve the problem you might be experiencing.

We hope we have been able to shed some light onto the mechanics working under hood, please let us know if this makes any sense to you. Regarding your issue with DEPATISnet, we will be happy to help further after gathering more information about your specific problem.

With kind regards,
Andreas.

[ElvezPelvez]

Andreas,

Thanks a lot for the quick reply. I'll apply for OPS access and will get back to you again.

Regarding your question: Yes, the two widgets are present and (if I remember correctly) I can get rid of the EPO one if I'm using "datasources = depatisnet, depatech" instead of "datasources = ops, depatisnet, depatech" in patzilla.ini (I'm running PatZilla using Docker btw.).

Best regards and thanks again,

Erik

[ElvezPelvez]

Hi Andreas,

I now have OPS acces, pasted the two keys into patzilla.ini (either with or without the { }) and restarted the container. I then tried the "Numberlist" and also the "Comfort Search".

However, I am getting the following error:

An exception occurred while processing your query.

Reason: pyramid.httpexceptions.HTTPBadGateway: Could not connect to OPS servers.

Any ideas what I might be doing wrong?

Thanks and best regards,

Erik

[ElvezPelvez]

The following is a log of the error - hope this helps...
I did check the access token on the EPO webpage and it worked alright.

E

date	stream	content
2019-02-11 17:07:39	stdout	2019-02-11 17:07:38,972 WARNING  [patzilla.access.epo.ops.client          ][waitress] Invalidating token and closing connection for client_id=xxxx
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,972 ERROR    [patzilla.access.epo.ops.client          ][waitress] OpsOAuth2Session HTTPError: 401 Client Error: Unauthorized for url: https://ops.epo.org/3.2/auth/accesstoken. client_id=xxxx
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,705 WARNING  [patzilla.access.epo.ops.client          ][waitress] Invalidating token and closing connection for client_id=xxxx
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,705 ERROR    [patzilla.access.epo.ops.client          ][waitress] OpsOAuth2Session HTTPError: 401 Client Error: Unauthorized for url: https://ops.epo.org/3.2/auth/accesstoken. client_id=xxxx
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,399 INFO     [mongodb_gridfs_beaker                   ][waitress] [MongoDBGridFS] Host URI: mongodb://mongodb:27017
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,349 INFO     [patzilla.navigator.services.ops         ][waitress] query finished
2019-02-11 17:07:38	stdout
2019-02-11 17:07:38	stdout	HTTPBadGateway: Could not connect to OPS servers.
2019-02-11 17:07:38	stdout	raise error
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/patzilla/access/epo/ops/client.py", line 190, in request
2019-02-11 17:07:38	stdout	return self.request('GET', url, **kwargs)
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
2019-02-11 17:07:38	stdout	response = client.get(url, headers={'Accept': 'application/json'}, params={'q': query, 'Range': range})
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/patzilla/access/epo/ops/api.py", line 223, in ops_published_data_search_real
2019-02-11 17:07:38	stdout	return ops_published_data_search_real(constituents, query, range)
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/patzilla/access/epo/ops/api.py", line 208, in ops_published_data_search
2019-02-11 17:07:38	stdout	return func(*args, **kwargs)
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/beaker/cache.py", line 597, in go
2019-02-11 17:07:38	stdout	v = self.createfunc()
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/beaker/container.py", line 378, in get_value
2019-02-11 17:07:38	stdout	return self._get_value(key, **kw).get_value()
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/beaker/cache.py", line 322, in get
2019-02-11 17:07:38	stdout	return cache[0].get_value(cache_key, createfunc=go)
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/beaker/cache.py", line 599, in cached
2019-02-11 17:07:38	stdout	result = ops_published_data_search(constituents, search.expression, range)
2019-02-11 17:07:38	stdout	File "/usr/lib/python2.7/site-packages/patzilla/navigator/services/ops.py", line 105, in ops_published_data_search_handler
2019-02-11 17:07:38	stdout	Traceback (most recent call last):
2019-02-11 17:07:38	stdout	exception:
2019-02-11 17:07:38	stdout	None
2019-02-11 17:07:38	stdout	response:
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,349 CRITICAL [patzilla.navigator.services             ][waitress] ops-search error: query="pn=EP0906861B1", reason=pyramid.httpexceptions.HTTPBadGateway: Could not connect to OPS servers.
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,340 WARNING  [patzilla.access.epo.ops.client          ][waitress] Invalidating token and closing connection for client_id=xxxx
2019-02-11 17:07:38	stdout	2019-02-11 17:07:38,340 ERROR    [patzilla.access.epo.ops.client          ][waitress] OpsOAuth2Session HTTPError: 401 Client Error: Unauthorized for url: https://ops.epo.org/3.2/auth/accesstoken. client_id=xxxx
2019-02-11 17:07:37	stdout	2019-02-11 17:07:37,106 INFO     [patzilla.navigator.services.ops         ][waitress] query cql: pn=EP0906861B1
2019-02-11 17:07:36	stdout	2019-02-11 17:07:36,785 INFO     [patzilla.util.expression                ][waitress] Parsing search expression "pn=EP0906861B1" with syntax "cql" and grammar "default"
2019-02-11 17:07:36	stdout	2019-02-11 17:07:36,784 INFO     [patzilla.navigator.services.ops         ][waitress] query raw: pn=EP0906861B1
2019-02-11 17:06:59	stdout	2019-02-11 17:06:59,704 INFO     [patzilla.access.epo.ops.client          ][waitress] OpsOAuthClientFactory.create_session: identifier=system, client_id=xxxx
2019-02-11 17:06:48	stdout	Serving on http://0.0.0.0:9999
2019-02-11 17:06:48	stdout	Starting server in PID 1.
2019-02-11 17:06:44	stdout	2019-02-11 17:06:44,844 INFO     [patzilla.access.epo.ops.client          ][MainThread] Creating OpsClientPool
2019-02-11 17:06:42	stdout	2019-02-11 17:06:42,154 INFO     [patzilla.util.config                    ][MainThread] Effective configuration files: /patzilla.ini
2019-02-11 17:06:42	stdout	2019-02-11 17:06:42,148 INFO     [patzilla.util.config                    ][MainThread] Expanded configuration files:  /patzilla.ini, /vendors.ini
2019-02-11 17:06:42	stdout	2019-02-11 17:06:42,142 INFO     [patzilla.util.config                    ][MainThread] Requested configuration files: /patzilla.ini
2019-02-11 17:06:42	stdout	2019-02-11 17:06:42,142 INFO     [patzilla.navigator.settings             ][MainThread] Root configuration file is /patzilla.ini

[amotl]

Thanks Erik,

regarding the error message

HTTPBadGateway: Could not connect to OPS servers.

The current problem you are seeing is on the code, sorry for that! I feel sad that you are experiencing things like that right now when just starting with PatZilla while the OPS interface was pretty much stable across the board over the last years ;].

After people already reported this to us (see #9 ff.), we have been able to partly fix this in the epo-ops-client branch but are currently working on a new release which also fixes some other issues introduced recently.

So, would you mind switching over to the branch if you feel adventurous in the meanwhile? Regarding the next release, we kindly ask for your patience. However, we will try to move this forward in the next few hours.

With kind regards,
Andreas.

[ElvezPelvez]

Hi Andreas and thanks a lot!

I'd love to try the branch and I will try to with to it even though I have not done tis before (quite new to GIT). Worst case I just delete the master branch ;-)

Best regards,

Erik

[ElvezPelvez]

Just a quick question: how would I switch to the epo-ops-client branch and rebuild the docker containers using the changed code?

Just do a:
git checkout epo-ops-client followed by a
docker-compose up

Thanks again,

Erik

[amotl]

a) Assuming you got things going on your workbench: Great to hear and thanks for sharing. @aghster will also be happy to hear that the docker environment works without any efforts even when switching branches.

b) If everything actually works well when accessing OPS, searching at DEPATISnet probably also works for you now?

[ElvezPelvez]

Sorry, but docker is not working. I'm still trying to figure out why: my two cents (I'm not too experienced) would be that the following line in Dockerfile causes the problem:

   pip install patzilla

This actually downloads patzilla-0.161.1.tar.gz from pythonhosted.org and does not care about the local repository, i.e. the ego-ops-client branch.

Since I really am not too experienced with either pip or docker I just hope that @aghster figures out a way to change the Dockerfile (or I just have to wait for your next release).

Thanks anyways and have a nice rest of the day,

Erik

[amotl]

Thanks for your feedback @ElvezPelvez.

Sorry, but docker is not working, [it] downloads patzilla-0.161.1.tar.gz from pythonhosted.org and does not care about the local repository, i.e. the ego-ops-client branch.

That's right, thanks for your investigation. As I am not using Docker with my development environment, I haven't been exactly aware of that.

I just hope that @aghster figures out a way to change the Dockerfile

In principle, I consider the current way a good thing as Docker users will get a stable release version for actually just running PatZilla without any effort on any environment supported by Docker. However, it would definitively be a cool thing if we also support a Docker-based development sandbox, so I've split the discussion about this topic into #10.

or I just have to wait for your next release

I fear it's really just on us this time to cut a new release, sorry again. If you feel more adventurous to setup a development environment directly on your workstation though, you might want to have a look at the corresponding sandbox setup documentation.

[ElvezPelvez]

Thanks a lot and I totally agree the Docker solution "as is" should be the best solution for users who want a stable release. However, I also like the idea of a sandboxed Docker version...

As long as no new release is out I will try the sandboxed setup as proposed and hope I will manage to get through the installation steps.

Thanks again for your help and effort!

[aghster]

The issue explained in #9 seems to be related to a python update (@amotl, did I get this right?). Therefore, a preliminary solution for using the current Docker production setup could be using an older python version. @ElvezPelvez, you could try replacing lines 4 and 5 in Dockerfile with:

  python2=2.7.14 \
  python2-dev=2.7.14 \

Unfortunately, right now I don't have the time to check myself whether this acutally works.

[aghster]

Now I've tried to at least reproduce the error, but everything runs just fine. Then I found that patzilla version 0.162.0 has been released on PyPi about 4 hours ago. So I suppose there is no need anymore for the above workaround, as @amotl apparently fixed #9.

[amotl]

Good to hear that everything works for you with the new release, enjoy your research!

Thanks also @aghster for suggesting the Python downgrade workaround. While this might have helped, let's just move forward.

[amotl]

Closing this now. Thanks again.