buddycloud / buddycloud-http-api Goto Github PK

It seems as if the session-related test cases in the test suite are broken. @abmargb Simon said you reorganized the session handling code, so I'm assigning to you.

Right now it is not practical to restart the HTTP API server in production settings because doing so causes all clients' sessions to be lost. It would be better if sessions were stored in a database or memcached. This would also make API server crashes less noticeable.

When the HTTP API opens a session for a user, it sends 2 <presence/> stanzas: one is a broadcast, the other is only to the channel domain.
I'm pretty sure the broadcast one is useless. But it causes the session to become visible in regular Jabber clients (tested with Gajim on buddycloud.org), which is annoying (and did not happen with the old webclient).

Any ideas why we receive an error here: https://demo.buddycloud.org/[email protected]

https://api.buddycloud.org/recommendations?max=5&user=testreg123%40buddycloud.org

Example:
1 Register '[email protected]' on demo.buddycloud.org
2 GET https://api.buddycloud.org/[email protected]/metadata/posts returns 404 - Not Found

On my server I get:

Error: getaddrinfo ENOENT
    at errnoException (dns.js:31:11)
    at Object.onanswer [as oncomplete] (dns.js:123:16)

When making a request to https://buddycloud.evilprofessor.co.uk/api/[email protected]/content/posts

I have proxied the API as follows:

RewriteRule ^/api/ http://localhost:3000/ [P,L]

Perhaps I am missing something with my proxy call?

When the user is not running services such as the 'channel-directory' service or the media server we really should be returning better error codes than a 500, e.g.

501 Not Implemented
502 Bad gateway
503 Service unavailable

i just updated my webclient/api-server on mightymonkey.net. the only thing that gets displayed after login is the sidebar. the posts/comments load for ever and are not displayed. (just the small "wheel").
i think this has something to do with the new "similar channels" feature. i am using search.buddycloud.com as suggested in the config.

here is my http-api log:

OUT xmpp: <iq type="get" from="[email protected]/0fdeb61c-7a38-4dae-a616-19bed2dada6a" to="search.buddycloud.com" id="66ad2d8abf6b109bf0057f1c1d11cfe6"><query xmlns="http://buddycloud.com/channel_directory/similar_channels"><channel-jid>[email protected]</channel-jid></query></iq>
IN xmpp: <iq id="66ad2d8abf6b109bf0057f1c1d11cfe6" type="error" to="[email protected]/0fdeb61c-7a38-4dae-a616-19bed2dada6a" from="search.buddycloud.com" xmlns:stream="http://etherx.jabber.org/streams"><error type="cancel"><remote-server-not-found xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/><text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">Server-to-server connection failed: host-unknown (This host does not serve search.buddycloud.com)</text></error></iq>

for full log: http://paste.kde.org/609272/

Just had to restart the server... Please include an init script similar to the buddycloud-server ( https://github.com/buddycloud/buddycloud-server/blob/master/_etc_init.d_buddycloud-server )

The API currently does not offer this feature if I am correct.

An additional feature would make it possible to replace an existing post or comment by a modified/edited version.

Maybe this helps somehow:
XEP-0308: Last Message Correction
http://xmpp.org/extensions/xep-0308.html

(The "Last" likely will be dropped in the next revision of that XEP.)

Hi do we consider adding authentication with Facebook, Google+, Twitter or OpenId?

On December 24th my API server crashed with the following at the end of the log:

timers.js:103
            if (!process.listeners('uncaughtException').length) throw e;
                                                                  ^
Error: Can't set headers after they are sent.
    at ServerResponse.OutgoingMessage.setHeader (http.js:645:11)
    at ServerResponse.res.setHeader (/opt/buddycloud-http-api/node_modules/express/node_modules/connect/lib/patch.js:62:20)
    at ServerResponse.res.header (/opt/buddycloud-http-api/node_modules/express/lib/response.js:280:8)
    at ServerResponse.res.send (/opt/buddycloud-http-api/node_modules/express/lib/response.js:85:10)
    at reportXmppError (/opt/buddycloud-http-api/src/util/api.js:91:7)
    at checkError (/opt/buddycloud-http-api/src/util/api.js:70:3)
    at exports.sendQuery (/opt/buddycloud-http-api/src/util/api.js:46:5)
    at Cache.Session._setupExpirationHandler._replyHandlers.onexpired (/opt/buddycloud-http-api/src/util/session.js:151:5)
    at Object._onTimeout (/opt/buddycloud-http-api/src/util/cache.js:49:12)
    at Timer.list.ontimeout (timers.js:101:19)

There was no activity in the log for several hours before that.

In the Atom items, author/uri is set by the bc server to acct:<JID of the poster>, but author/name is left untouched (see Content normalization in the XEP). This way clients can use a prettier name (your full name or a nickname) if you want.

But one should not trust author/name to be the real JID of the poster, nor to be a JID at all. If present, it may be used to print a pretty name, but for finding an avatar and displaying the real JID clients must use author/uri. So this really should be included in responses from the HTTP API. (I just posted an example in [email protected], as "Bad Atom forger" ;))

Instead of returning an empty list, the API is returning an response when there is no similar channels.

Users on servers hosting several domains need to use their full JID to login.

I'm not using a pusher component on my domain.

$ curl -I --basic --user [email protected] https://api.pouet.im/user_settings
HTTP/1.1 500 Internal Server Error

A quick workaround is to make getSettings() in src/user_settings.js send an empty JSON dictionary... but it's a little less than optimal.

When packaging, the config file needs to be read from /etc/buddycloud-http-api. Being able to run the server with a --config would make life easier.

i run my own server but use the bc.org searchComponent.

if i click on "find channels" it doenst load.
here is my http-api log for that action:

127.0.0.1 - - [Sat, 10 Nov 2012 19:25:10 GMT] "GET /most_active?max=5&user=test%40mightymonkey.net HTTP/1.1" 401 12 "https://mightymonkey.net/explore" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko) Ubuntu/12.10 Chromium/22.0.1229.94 Chrome/22.0.1229.94 Safari/537.4"
IN xmpp: Communication with remote servers is not allowed
127.0.0.1 - - [Sat, 10 Nov 2012 19:25:10 GMT] "GET /recommendations?max=5&user=test%40mightymonkey.net HTTP/1.1" 401 12 "https://mightymonkey.net/explore" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko) Ubuntu/12.10 Chromium/22.0.1229.94 Chrome/22.0.1229.94 Safari/537.4"

There's a little problem with Tigase clusters: the HTTP API need to connect to the same node as the channels server in order to be able to communicate with it.
Until it's fixed in Tigase, we should make it possible to have the HTTP API connect to a configurable IP as a workaround.

Hi,

I've been trying to set up the http api server. I've built it on the webhost I'm using, and started it with npm start. I've been testing it by sending the following request:

[highfellow@web225 ~]$ curl -v --basic -u [email protected]:xxx localhost:22024/[email protected]/content/posts?max=1
* About to connect() to localhost port 22024
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 22024
* Server auth using Basic with user '[email protected]'
> GET /[email protected]/content/posts?max=1 HTTP/1.1
> Authorization: Basic *(removed)*
> User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: localhost:22024
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection #0

The server contacts the xmpp server, which replies with a response, at which point the http server seems to crash:

[highfellow@web225 buddycloud-http-api]$ npm start

> [email protected] start /home/highfellow/build/buddycloud-http-api
> node ./server.js

Cannot load StringPrep-0.1.0 bindings. You may need to `npm install node-stringprep'
Warning: program compiled against libxml 209 using older 206
Server started with configuration profile "production"
Listening on port 22024

OUT xmpp: <iq type="get" from="[email protected]/220069a8-cff7-488d-acbf-821833740bec" to="buddycloud.highfellow.org" id="f341a31022293a4cc62840bbc7cfe110"><pubsub xmlns="http://jabber.org/protocol/pubsub"><items node="/user/[email protected]/posts" max_items="1"/><set xmlns="http://jabber.org/protocol/rsm"><max>1</max></set></pubsub></iq>
IN xmpp: <iq id="f341a31022293a4cc62840bbc7cfe110" type="result" to="[email protected]/220069a8-cff7-488d-acbf-821833740bec" from="buddycloud.highfellow.org" xmlns:stream="http://etherx.jabber.org/streams"><pubsub xmlns="http://jabber.org/protocol/pubsub"><items node="/user/[email protected]/posts"><item id="4d92da42-aaa1-4374-80b5-dce5d5575809"><entry xmlns="http://www.w3.org/2005/Atom"><content>Glad you like it :-)</content><author><name>[email protected]</name><uri>acct:[email protected]</uri></author><in-reply-to xmlns="http://purl.org/syndication/thread/1.0" ref="6376aa0d-5d95-4fa1-a37f-976ac429113e"/><id>4d92da42-aaa1-4374-80b5-dce5d5575809</id><published>2012-10-13T00:51:01.442Z</published><updated>2012-10-13T00:51:01.442Z</updated><link rel="self" href="xmpp:buddycloud.highfellow.org?pubsub;action=retrieve;node=/user/[email protected]/posts;item=4d92da42-aaa1-4374-80b5-dce5d5575809"/><verb xmlns="http://activitystrea.ms/spec/1.0/">post</verb><object xmlns="http://activitystrea.ms/spec/1.0/"><object-type>comment</object-type></object></entry></item></items><set xmlns="http://jabber.org/protocol/rsm"><first index="0">4d92da42-aaa1-4374-80b5-dce5d5575809</first><last>4d92da42-aaa1-4374-80b5-dce5d5575809</last><count>590</count></set></pubsub></iq>
[highfellow@web225 buddycloud-http-api]$ 

The response from the xmpp server is never returned as an http request (see the first pasted section).

The only thing I remember doing differently when built the server was that I downgraded the version of node-stringprep to 0.1.4 to get it to build (with 0.1.5 it gave a build error).

If you have any ideas what is happening, or how to debug this further, that would be helpful.

Cheers,

[email protected]

As per title (see post on [email protected])

Show following events to the client when they happen:

https://trello.com/card/following-requests/504114227dc8c2e35d039db6/42

New POSTs are not supposed to have a replyTo attribute, but GET /notifications/posts is returning the last POST from a channel as the replyTo from a previous one.

It's causing: buddycloud/webclient#79

If user is not running a media server, or media server is not running for some reason then rather than respond with a 503 then HTTP API should return 302 to a gravatar URL.

Actually this should be a config option (useGravatarFallback or something).

This prevents the build up of lots of identical avatars and helps quick identification of users in channels.

At this moment, the API is simply logging to the console. We need a better logging mechanism that should be able to write to a file, including timestamps and different log levels.

Seems that GET /subscribed is returning 100 elements max as result.

18:10
Schnouki
yep, the http api and new webclient don't know about /status nodes at all

18:11
Simon
we should really have the API server as node-agnostic: /:channel/content/

Apigee makes it really simple to play with APIs.

This is a simple task for someone getting into buddycloud: add buddycloud's API to a page on Apigee like https://apigee.com/console/twitter

If we try to create an already existent node in a channel, the API returns "Internal Server Error".

Add an extra field called mentions or mentionsCount, with the count of posts with user's mentions (per channel).

See buddycloud/deprecated-buddycloud-server#108

@rjawahar
"Hi,
Checked out the demo site and tried creating a new private topic channel and it just hangs saying 'creating' and didnt complete."

Sometimes, when a new post/comment is created, the response from the HTTP API is like:

{"id":"c3474716-8878-45dd-af89-540f1b4823f7","author":"[email protected]","published":null,"updated":null,"content":"buggy post"}

Posted here because it is the server used by the webclient, but this can be a bc-server issue.

A possible future functional test suite would have to be able to delete accounts which were created for or during test cases. Thus, it would be nice if the HTTP API would allow this with DELETE /account, even if only for admins.

The XEP documentation
https://buddycloud.org/wiki/XMPP_XEP#Create_a_post

"Clients (especially in Web browsers) SHOULD NOT display rich content of entry/content[@type='html' or @type='xhtml'] without sanitizing markup and scrubbing JavaScript".

Question: are content types html and xhtml currently supported by the API ?

Right now the apache log is filled with CORS preflight checks/HTTP OPTIONS requests.

The API server needs a timeout of ~1 day: http://www.w3.org/TR/cors/#http-access-control-max-age

To check, pull up:

curl -H "Origin: http://example.com" --verbose https://api.buddycloud.org/

Looks like there is a memory lear in the API server.

Currently using 1.6GB virtual memory and 775MB resident memory on a lightly loaded demo.buddycloud.org

This seems like a lot of memory for a few concurrent sessions.

I've been seeing the following warning message a lot lately (on crater, in the log file):

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Client.<anonymous> (events.js:139:15)
    at Client.once (events.js:160:8)
    at Session.onStanza (/opt/buddycloud-http-api/src/util/session.js:254:20)
    at listenForNextItem (/opt/buddycloud-http-api/src/notifications_posts.js:39:15)
    at callbacks (/opt/buddycloud-http-api/node_modules/express/lib/router/index.js:272:11)
    at provideSession (/opt/buddycloud-http-api/src/util/session.js:61:3)
    at /opt/buddycloud-http-api/src/util/session.js:47:7
    at callbacks (/opt/buddycloud-http-api/node_modules/express/lib/router /index.js:272:11)
    at param (/opt/buddycloud-http-api/node_modules/express/lib/router/index.js:246:11)
    at pass (/opt/buddycloud-http-api/node_modules/express/lib/router/index.js:253:5)

Should we do something about it?

If I use username "tuomas" instead of "[email protected]", I seem to receive 500 Internal Server Error

How about sending 401 Unauthorized or 403 Forbidden ?

When scrolling down on a private channel in the webclient, eventually the API uses an anonymous connection to fetch posts of the channel, which obviously leads to a "Requires affiliation member (you are none)" error.

Traces:
// First request for posts
OUT xmpp: <iq type="get" from="[email protected]/da70e837-461e-4b14-906b-8ac0da57a653" to="channels.buddycloud.org" id="cca93122bdf7065ce48afe2e0852c155"><pubsub xmlns="http://jabber.org/protocol/pubsub"&gt;&lt;items node="/user/[email protected]/posts" max_items="51"/><set xmlns="http://jabber.org/protocol/rsm"&gt;&lt;max&gt;51&lt;/max&gt;&lt;/set&gt;&lt;/pubsub&gt;&lt;/iq>
// Successfully replied
IN xmpp: <iq id="cca93122bdf7065ce48afe2e0852c1 ....

// Second request for posts
OUT xmpp: <iq type="get" from="[email protected]/e7382733-4f27-46ca-965b-bff031dc6de8" to="channels.buddycloud.org" id="b90d9c078c24abcabc5b6a5fa98b9483"><pubsub xmlns="http://jabber.org/protocol/pubsub"&gt;&lt;items node="/user/[email protected]/posts" max_items="51"/><set xmlns="http://jabber.org/protocol/rsm"&gt;&lt;max&gt;51&lt;/max&gt;&lt;after&gt;ae3fdb09-a345-42c7-bae4-e6f86d56789f&lt;/after&gt;&lt;/set&gt;&lt;/pubsub&gt;&lt;/iq>
// Replied with error
IN xmpp: <iq id="b90d9c078c24abcabc5b6a5fa98b9483" type="error" to="[email protected]/e7382733-4f27-46ca-965b-bff031dc6de8" from="channels.buddycloud.org" xmlns:stream="http://etherx.jabber.org/streams"&gt;&lt;error type="auth"><forbidden xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/><text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">Requires affiliation member (you are none)</text></error></iq>

Actually that's a backend problem.

The return message should be more user friendly.

Ex: http://api.buddycloud.org/search?type=abc&q=test

In config I have media server URL commented out (so config key would be undefined) as I'm not running one. I therefore see lots of this in the http-api server logs

TypeError: Parameter 'url' must be a string, not undefined
    at Object.urlParse [as parse] (url.js:96:11)
    at getMediaUrl (/opt/buddycloud-http-api/src/media.js:63:22)
    at proxyToMediaServer (/opt/buddycloud-http-api/src/media.js:51:18)
    at callbacks (/opt/buddycloud-http-api/node_modules/express/lib/router/index.js:272:11)
    at exports.mediaServerDiscoverer (/opt/buddycloud-http-api/src/util/api.js:199:5)
    at /opt/buddycloud-http-api/src/util/dns.js:42:7
    at asyncCallback (dns.js:68:16)
    at Object.onanswer [as oncomplete] (dns.js:149:9)

Running against a Tigase environment:

/usr/share/buddycloud-http-api/src/notifications_posts.js:68
var entry = items.get('p:item/a:entry', {
^
TypeError: Cannot call method 'get' of undefined
at extractItem (/usr/share/buddycloud-http-api/src/notifications_posts.js:68:21)
at /usr/share/buddycloud-http-api/src/notifications_posts.js:42:18
at Client.Session.onStanza (/usr/share/buddycloud-http-api/src/util/session.js:257:5)
at Client.g (events.js:192:14)
at Client.EventEmitter.emit (events.js:126:20)
at Client.onRawStanza (/usr/share/buddycloud-http-api/node_modules/node-xmpp/lib/xmpp/client.js:159:14)
at Client.EventEmitter.emit (events.js:96:17)
at Client.Connection.onStanza (/usr/share/buddycloud-http-api/node_modules/node-xmpp/lib/xmpp/connection.js:237:14)
at StreamParser.Connection.startParser (/usr/share/buddycloud-http-api/node_modules/node-xmpp/lib/xmpp/connection.js:136:14)
at StreamParser.EventEmitter.emit (events.js:96:17)

If I do this:

curl -v --data '{"content":"Hello & world"}' https://tuomas%40buddycloud.org:[email protected]/[email protected]/posts

only "hello" is posted. Rest disappears.

Right now, /sync just syncs the POSTS node.

The URL would be like: api.buddycloud.org/:node/sync

A little earlier today the http api server on crater had 767 open C2S connections to Prosody (as per netstat -anpe | grep <http-api-PID> | grep ':5222' | wc -l). Because of that Prosody went spinning and had to be restarted.

I'm not sure what's causing this -- could it be failed anon connections that aren't expired correctly? Anyway, this has already happened before. I'll try to grab more details from the log file next time.

Earlier today, after we had to restart Prosody on crater, the http api server started to use a lot of CPU for a few minutes. There were many error messages in log: some about the XMPP <stream/> being closed, and others "unable to call write on undefined" (or something like this) with stack traces.
(Sorry I can't be more specific, the http api server was then restarted and its log was truncated before I could copy-paste it here...)

I suspect the stack traces are from cached sessions being invalidated when the XMPP server becomes unavaiable, causing trouble later. Would it be possible to detect such cases and fail more "cleanly"?

Hi Denis,

Been trying to debug a 500 error message and am unable to find logs.

Please log to /var/log/buddycloud-http-api.log

It would filter results for a single channel jid.