Looking at the code it seems like you're including the XML-RPC parameters in the signature base string? I'm certainly no master at Python so I may have misunderstood it - but if I haven't then it would most certainly be the wrong behavior.
The parameters of the body should only be included in the signature base string if the content type of the POST-request was application/x-www-form-urlencoded according to the spec: #sig" target="_new">http://oauth.net/core/1.0/#signormparam
@voxpelli: we are just shivving xml-rpc into the oauth flow, we are not using an xml-rpc extension, we treat parameters coming via xml-rpc as plain POST params, we are not going to use body hashing for it
@termie: I understand - but if the request's body isn't encoded as application/x-www-form-urlencoded then those parameters should not be included in the signature base string. If they are - as I think this patch would make them - the signature will not match the one sent in the request
@voxpelli: when a request comes in to a web app the parameters have already been decoded, the oauth library re-encodes them to generate the base string and signature, there is no difference where the parameters came from, as long as you have the actual value of the parameter (which in all cases you do otherwise you would not be making the request you think you are making) you are able to generate a proper signature
@termie: I know - but the OAuth specification says otherwise - look in section 9.1.1 in the specification: http://oauth.net/core/1.0/
It's a flaw in the specification - I'm involved in building an API with a non-supported content type on the request body as well and we was also surprised by this fact.
There are two solutions - one is to not verify the integrity - another is to use the body hash extension which Brian Eaton at Google wrote together with the OAuth community for use with OpenSocial.
@voxpelli: there are plenty of content types for which this does not apply and body signing is the only reasonable way to verify the integrity of the content (atom publishing protocol stuff, for example), however in this case, based on the restrictions we place on how the xml-rpc interface is used, the only things needing verification are the values of the parameters themselves, they are passed in as a dictionary style object and we are not using positional args.
And, for the record, it is not a flaw in the specification, we chose to omit body signing from the scope of the first version and leave it for future extensions to handle.
@termie: ok - apparently you know much more about this than me - didn't know you was one of the authors behind the specification - so I wont question you on this one
Just out of curiosity though - how are you interpreting 9.1.1 into allowing this? Doesn't it specifically demand a form-encoded body? I looked at Eran's draft of the editor's cut now as well and can find no other interpretation there either - if not all of the params is sent in XML-RPC through some kind of extension to the protocol which would be allowed?
@voxpelli: well first off using xml-rpc obviously isn't covered in the spec, the spec only talks about headers and post and get, what we are doing however is following the spec for signing a set of request parameters.
A quick walkthrough of signing a request based on parameters of some sort (have to break it into multiple comments):
Step 1: Get the parameters. The spec describes three sources for parameters, the header, POST application/x-www-form-urlencoded and GET query string.
The reason it specifies a content type there is to explicitly say this is data expected as key-value pairs, no mime attached files, no atompub, only things that are made of the same kind of key-values as one would expect in a query string.
In our case we choose to also allow those key-value pairs to come in via xml-rpc. When you boil it all down, what you are trying to sign is the important part of the request, in atompub the organization of the actual xml can have significance, tags will have attributes, etc; however, in xml-rpc (when used the way we are using it) the "information," the important part of the request, is just the key-value pairs, we completely ignore all that extra xml that makes up the xml-rpc protocol: all we want is the content.
Once you have those key-value pairs you need to apply predictable and repeatable transformations to it to make sure that each key and value looks exactly the same as it does to the person who sent them, in the case of oauth that is a specific type of urlencoding.
Regardless of where you got the variables from, be it POST, GET, headers, xml-rpc, you will have a key and a value and your web app will have successfully decoded them from whatever format they were in to the actual values of the key and the value without any escaping, that is inherently necessary if you plan on using those values to make changes to your site. From there it is a simple matter of allowing the OAuth library to normalize them for use in the signature.
When writing the oauth libraries, I explicitly included the option of passing a parameters dict or associative array into the request constructors so that the libraries can be transport protocol agnostic.
Step 3 and onward: I'm pretty confident you've got the rest from there.
@termie: Thanks for a great explanation - got the feeling that it was something like that when I read the editor's cut - the original specification left an impression on me that it wasn't allowed
Makes much more sense now - felt like a flaw when the spec was interpreted as only allowing parameters in those three ways - now I understand what you meant.
So this is LGTM for me as well - was fun to dig into some python code
20 comments so far
http://rietku.appspot.com/16003
6 months ago by guywithkeyboard
LGTM :)
6 months ago by lemonad
@lemonad: had to lookup that... "looks good to me"
6 months ago by BUGabundo
@BUGabundo: Yes, I had to Google that the first time I saw it too :)
6 months ago by lemonad
Looking at the code it seems like you're including the XML-RPC parameters in the signature base string? I'm certainly no master at Python so I may have misunderstood it - but if I haven't then it would most certainly be the wrong behavior.
The parameters of the body should only be included in the signature base string if the content type of the POST-request was application/x-www-form-urlencoded according to the spec: #sig" target="_new">http://oauth.net/core/1.0/#signormparam
The core oAuth specification doesn't itself deal with verifying the integrity of an XML-RPC requests body - for that you would need to use the body hash extension: http://oauth.googlecode.com/svn/spec/ext/body_hash/1.0/oauth-bodyhash.html
6 months ago by voxpelli
@voxpelli: Thanks a lot for reviewing! I added a reference to your comment to rietku!
6 months ago by lemonad
@voxpelli: we are just shivving xml-rpc into the oauth flow, we are not using an xml-rpc extension, we treat parameters coming via xml-rpc as plain POST params, we are not going to use body hashing for it
6 months ago by termie
@termie: I understand - but if the request's body isn't encoded as application/x-www-form-urlencoded then those parameters should not be included in the signature base string. If they are - as I think this patch would make them - the signature will not match the one sent in the request
6 months ago by voxpelli
@voxpelli: when a request comes in to a web app the parameters have already been decoded, the oauth library re-encodes them to generate the base string and signature, there is no difference where the parameters came from, as long as you have the actual value of the parameter (which in all cases you do otherwise you would not be making the request you think you are making) you are able to generate a proper signature
6 months ago by termie
@termie: I know - but the OAuth specification says otherwise - look in section 9.1.1 in the specification: http://oauth.net/core/1.0/
It's a flaw in the specification - I'm involved in building an API with a non-supported content type on the request body as well and we was also surprised by this fact.
There are two solutions - one is to not verify the integrity - another is to use the body hash extension which Brian Eaton at Google wrote together with the OAuth community for use with OpenSocial.
6 months ago by voxpelli
@voxpelli: there are plenty of content types for which this does not apply and body signing is the only reasonable way to verify the integrity of the content (atom publishing protocol stuff, for example), however in this case, based on the restrictions we place on how the xml-rpc interface is used, the only things needing verification are the values of the parameters themselves, they are passed in as a dictionary style object and we are not using positional args.
And, for the record, it is not a flaw in the specification, we chose to omit body signing from the scope of the first version and leave it for future extensions to handle.
6 months ago by termie
@termie: ok - apparently you know much more about this than me - didn't know you was one of the authors behind the specification - so I wont question you on this one
Just out of curiosity though - how are you interpreting 9.1.1 into allowing this? Doesn't it specifically demand a form-encoded body? I looked at Eran's draft of the editor's cut now as well and can find no other interpretation there either - if not all of the params is sent in XML-RPC through some kind of extension to the protocol which would be allowed?
6 months ago by voxpelli
@voxpelli: well first off using xml-rpc obviously isn't covered in the spec, the spec only talks about headers and post and get, what we are doing however is following the spec for signing a set of request parameters.
A quick walkthrough of signing a request based on parameters of some sort (have to break it into multiple comments):
6 months ago by termie
Step 1: Get the parameters. The spec describes three sources for parameters, the header, POST application/x-www-form-urlencoded and GET query string.
The reason it specifies a content type there is to explicitly say this is data expected as key-value pairs, no mime attached files, no atompub, only things that are made of the same kind of key-values as one would expect in a query string.
In our case we choose to also allow those key-value pairs to come in via xml-rpc. When you boil it all down, what you are trying to sign is the important part of the request, in atompub the organization of the actual xml can have significance, tags will have attributes, etc; however, in xml-rpc (when used the way we are using it) the "information," the important part of the request, is just the key-value pairs, we completely ignore all that extra xml that makes up the xml-rpc protocol: all we want is the content.
6 months ago by termie
Step 2: Normalize the parameters
Once you have those key-value pairs you need to apply predictable and repeatable transformations to it to make sure that each key and value looks exactly the same as it does to the person who sent them, in the case of oauth that is a specific type of urlencoding.
Regardless of where you got the variables from, be it POST, GET, headers, xml-rpc, you will have a key and a value and your web app will have successfully decoded them from whatever format they were in to the actual values of the key and the value without any escaping, that is inherently necessary if you plan on using those values to make changes to your site. From there it is a simple matter of allowing the OAuth library to normalize them for use in the signature.
When writing the oauth libraries, I explicitly included the option of passing a parameters dict or associative array into the request constructors so that the libraries can be transport protocol agnostic.
Step 3 and onward: I'm pretty confident you've got the rest from there.
6 months ago by termie
@termie: Thanks for a great explanation - got the feeling that it was something like that when I read the editor's cut - the original specification left an impression on me that it wasn't allowed
Makes much more sense now - felt like a flaw when the spec was interpreted as only allowing parameters in those three ways - now I understand what you meant.
So this is LGTM for me as well - was fun to dig into some python code
6 months ago by voxpelli
decided this might make a good blog post so I copied it over to http://term.ie/devdev/oauth-via-xmlrpc
6 months ago by termie
wow looks like i missed the party - only if there were an android client...
6 months ago by guywithkeyboard
@guywithkeyboard: Or some kind of personalised Explore panel...
6 months ago by adewale
@guywithkeyboard: there is a bit of a mobile site now, for the record: m.jaiku.com
6 months ago by termie