PayPal IPN Failures

[railer]

Having problems with our site not receiving the PayPal IPN notices. It was working up until 4/12/2019 but seemed to have stopped then. Haven't been able to come up with a reason, though I have been implementing a lot of IP range bans in cpHulk due to a large number of failed login attempts. I am also using fail2ban with dovecot filters.

Would that affect PayPal?

Sandbox transactions work fine.

Anyway, I would like for you guys to take a look. Can't see a way to purchase a Pro subscription. Please advise on how to accomplish this.

For your info, here is the error that is coming in the fabrik logs:

/index.php?option=com_fabrik&c=plugin&task=plugin.pluginAjax&formid=2&g=form&plugin=paypal&method=ipn&renderOrder=0
-----------------------------------------
"form.paypal.ipnfailure.custom_error"
--------------------------

//////////////

//////////////
cmd=_notify-validate
//////////////
formid or rowid empty in custom:
IPN custom function =
IPN custom transaction function =

---------------

And I've attached a screen capture of the form's PayPal plugin config.

 

[cheesegrits]

Looks like you are getting the IPN callback, but no request variables are making it through to Fabrik.

Is there a 'fabrik.ipn.start' log?

Re Pro support, we don't do "all you can eat" subscriptions any more, just hourly rate.

-- hugh

 

[railer]

Hi Hugh,

Sorry about the delay in responding. Just now getting back to this issue. I have separately submitted a RFQ (2019-05-28) and hope to hear from you soon.

Yes, there is a fabrik.ipn.start log. I'm not quite sure what records I'd need to copy info from and paste here, but I have an .xlsx file if you care to look at the entries, but can't upload the .xlsx file. Just let me know where I can send it.

FYI, there are several message types in the log:

• fabrik.ipn.start
• fabrik.paypal.onAfterProcess
• form.paypal.ipnfailure.custom_error

At another time, there was this:
• form.paypal.ipndebug.ipn_query

There are several plugins in the form:
• PayPal: - processes the payment
• Redirect - employer to job confirm page — This is unpublished (Instead, we have a message in the PP plugin After Payment tab - see uploaded screen capture file. Going to delete this plugin.)
• email: to staff member
• email: confirmation to employer
• php: On edit - checks to see if payment was successful or not (employer might not have completed form on first try).
• Paypal: on edit - if payment incomplete redirect to Paypal
• php: displays thank you message
• php: saves status as 1 if staff overrides incomplete payment

QUESTION: I'm not really clear on what URL should be entered into the client's IPN Notification setting in their PayPal account? I currently just have it going to the URL of a Joomla page with a Job Confirmation message on it. Does it need any special coding related to it to receive that data?
My developer says that the payment works in Sandbox test mode.

PS: I let Robbie know about this and I believe she mentioned this to you.

 

[railer]

Noted the following in an old post:

"So ... in your "IPN Custom Value", put the {tablename___item_id_raw} placeholder, and select a field to use as the IPN Custom Element. If necessary, create a simple field element for it. Let's assume it's called yourtable___ipn_custom."

Which we don't have defined in the Paypal plugin's IPN tab.

http://fabrikar.com/forums/index.php?threads/paypal-form-plugin-and-ipn.40934

My developer said we have defined the values for the transaction in the other fields in the IPN tab.

 

[railer]

Bumping this in hopes of a response.

On the question of what IPN URL to enter into my client's PayPal account, I'm going to assume that Fabrik sends a callback URL during the payment process. So I think I'd just need to enter the site's home page URL, as I've seen other ecommerce apps do.
Please advise if this is wrong.

That leaves the initial problem of why Fabrik isn't logging the callback.

I think that since this is a job listing site, that maybe the sheer length of data input into the form, maybe be causing problems if Fabrik sends that data, too? Your help/advice would be very much appreciated.

 

[railer]

I'm focusing my efforts on solving this missing info in the log message:

formid or rowid empty in custom:
IPN custom function =
IPN custom transaction function =

I do see the following text in the PayPal IPN message sent by PayPal. Note the "custom=2:247" part:

notify_version=3.9&custom=2:247:Completed

If you could possibly point me in the right direction as to what to do, I'd really appreciate it.

 

[railer]

After comparing the Fabrik log entries for successful text transactions with the failed live transactions, I noticed that both start with the following log message_type: fabrik.paypal.onAfterProcess

They contain what appear to be some parameters assembled by the PayPal plugin, along with all of the form data collected. In the tests, which are from a separate form with just a few field elements, the log entry's message text is very minimal. Whereas the "message" text for the actual job listing form has all of the input data in it for the job listing.

Is it likely that the sheer length of data in the job listing form may be bogging down the PayPal part of the process?

If so, what would be the procedure for only sending the information required by PayPal to finish the transaction?

 

[railer]

Would I be offbase to say that the following — taken from a successful transaction log — would be the necessary parameters required by PayPal? I'm think that since PayPal sent this back to me, that those are what it needs.

"option":"com_fabrik"
"c":"plugin"
"task":"plugin.pluginAjax"
"formid":"6"
"g":"form"
"plugin":"paypal"
"method":"ipn"
"renderOrder":"1"
"mc_gross":"1.00"
"protection_eligibility":"Eligible"
"address_status":"confirmed"
"payer_id":"IEKDIEKDKEIDK"
"address_street":"1 Main St"
"payment_date":"11:51:43 Jun 26, 2019 PDT"
"payment_status":"Completed"
"charset":"windows-1252"
"address_zip":"12345"
"first_name":"sss"
"mc_fee":"0.33"
"address_country_code":"US"
"address_name":"sss sss"
"notify_version":"3.9"
"custom":"6:18:"
"payer_status":"verified"
"business":"buyer@user.com"
"address_country":"United States"
"address_city":"Anytown"
"quantity":"1"
"verify_sign":"efsejklKLEskelKelsekaselkfneklKLekfjeks23kel4323jkl.yse"
"payer_email":"buyer@user.com"
"txn_id":"JeskUesmeIsBEsk"
"payment_type":"instant"
"last_name":"sss"
"address_state":"CA"
"receiver_email":"buyer@user.com"
"payment_fee":"0.33"
"shipping_discount":"0.00"
"insurance_amount":"0.00"
"receiver_id":"EJD(FJSN&SJF"
"txn_type":"web_accept"
"item_name":"check"
"discount":"0.00"
"mc_currency":"USD"
"item_number":""
"residence_country":"US"
"test_ipn":"1"
"shipping_method":"Default"
"transaction_subject":""
"payment_gross":"1.00"
"ipn_track_id":"bb484df8870f6"
"Itemid":null
"view":"plugin"

 

[cheesegrits]

Hey, sorry, I had to have surgery on a badly broken finger, I've been sidelined for a while.

I'm reading the thread now ...

-- hugh

 

[cheesegrits]

OK, in no particular order:

On the question of what IPN URL to enter into my client's PayPal account, I'm going to assume that Fabrik sends a callback URL during the payment process. So I think I'd just need to enter the site's home page URL, as I've seen other ecommerce apps do.

You don't need to enter anything. We supply the IPN callback URL in our initial redirect to the PayPal site, which directs their IPN call back to our plugin.

formid or rowid empty in custom:
IPN custom function =
IPN custom transaction function =

I do see the following text in the PayPal IPN message sent by PayPal. Note the "custom=2:247" part:

notify_version=3.9&custom=2:247:Completed

Hmm, well that's definitely the problem. But I don't quite see why.

Here's what that's about:

PayPal lets you (well, us) include a 'custom' value in the initial redirect, where you also tell it the IPN callback URL. It then passes that value back untouched as a query string arg to the IPN callback, which is the "&custom=6:18:" you are seeing. We use that to store the form ID and row ID of the record being submitted, so we know which row to update with the IPN info when the IPN callback arrives. We also allow you to specify your own custom value, which we add to that triple ... looks like you specified one with "&custom=2:247:Completed". So the triple is "<form id>:<row id>:<custom value>". If you don't specify a custom value, the third part is blank. If you do specify a custom value, we store the reflected value into the custom IPN field you specify. You can also use it if you have your own IPN handling code.

But the important part is that the form and row ID should be passed to PayPal (and back through IPN) regardless of whether you specify an IPN custom value. That's optional, just added to the triple. We always add them before the initial redirect:

https://github.com/Fabrik/fabrik/blob/master/plugins/fabrik_form/paypal/paypal.php#L452

If we can't extract the form ID and row ID from that 'custom' arg in the IPN callback processing, we log that 'form.paypal.ipnfailure.custom_error' and go no further. We grab and explode the colon separated custom triple here:

https://github.com/Fabrik/fabrik/blob/master/plugins/fabrik_form/paypal/paypal.php#L611

... and bang out if we couldn't find the form or rod ID here (can't remember why we don't bang out earlier) ...

https://github.com/Fabrik/fabrik/blob/master/plugins/fabrik_form/paypal/paypal.php#L716

But the strange thing is, if the list of args coming in from PayPal you showed in your last post is associated with a custom_error failure ... I don't see why, because the form ID and row ID were there in the &custom. ALthough re-reading your last msg, those are from a successful transaction?

What I need to see is that full list of args coming from paypal when you get the custom_error.

-- hugh

 

[railer]

Welcome back Hugh. Sorry to hear about your finger, but glad you had it worked on and thanks for your post above.

Those args in my post above are not from a custom_error failure. They're from a successful test.

Still struggling with this problem.

I've been checking my Apache logs to see if anything from PayPal is being blocked but I don't see anything. I've also read that mod_security can block IPN callbacks because they don't supply a User-Agent header, but I don't see any hits from PayPal in mod_security tools. This page at PayPal deals with how to set mod-security to accept PayPal IPN requests. Keeping that for later if needed.

 

[railer]

Update: I did some sandbox tests again, which were successful -- the IPN requests came through.

I used the 11.22 amount recommended at PayPal's IPN Sandbox testing page as a trigger for the "Pending" test. And it ended up sending these three message_types:

1. form.paypal.ipn.Pending
2. fabrik.ipn.start
3. form.paypal.ipndebug.ipn_que
4. form.paypal.ipn.Denied.

Who knows what that means? Haha, I gave up on those stupid things.

I skipped to a live test with a $0.01 payment amount. This test resulted in the following three log entries:

1. fabrik.paypal.onAfterProcess
2. fabrik.ipn.start
3. form.paypal.ipn.Completed

The test job listing came was processed on our end, changing its status from Pending to Active (published). So all appears to be good.
I'm still at a loss as to why it is working now. Though I am thinking that one thing which could be affecting it is that I changed the max_execution_time php.ini setting from 30 seconds to 90 seconds the other day. What do you think? Sometimes this site seems to run quite slow. FYI, this is one of two slave sites running in a JMS Multisite platform. I think the culprit may be a bunch of custom calc elements which perform various chores for the data. Not sure, though.

 

[cheesegrits]

I presume "form.paypal.ipndebug.ipn_que" was actual "form.paypal.ipndebug.ipn_query".

In which case, both of those log sequences look good (although the first one should have had an onAfterProcess one as well, as that's logged as the form is submitted, before the redirect to PayPal). The first one is basically ...

PayPal makes an IPN callback saying "Pending". We shrug, and do nothing (well, I think we update the status field, if you specified one).
At some point later, PayPal makes another IPN callback denying the payment, we update status again.

In the second case, a single IPN callback for success (Completed), we update status (although I would have expected a ipn_query one, unless you turned off test mode) and any other IPN fields you have specified (like amount, etc).

I'm still at a loss as to why it is working now. Though I am thinking that one thing which could be affecting it is that I changed the max_execution_time php.ini setting from 30 seconds to 90 seconds the other day. What do you think? Sometimes this site seems to run quite slow. FYI, this is one of two slave sites running in a JMS Multisite platform. I think the culprit may be a bunch of custom calc elements which perform various chores for the data. Not sure, though.

Heavy calcs and lots of data shouldn't affect IPN. It would effect initial form submission, but if it gets as far as redirecting to PayPal, all is good. In the IPN callback processing, we intentionally keep the processing to an absolute minimum. We do direct database access, to just poke values into the right row directly, rather than going through any of our form / element model stuff, so nothing like calc handling happens. The IPN handler does involve Joomla, it goes through the standard J! routing and dispatching, loading up the J! framework, but that's pretty quick.

I'm assuming you don't have your own IPN handler code (in plugins/fabrik_form/paypal/scripts/paypal_ipn.php)? That would be the only other place code would get run on an IPN response. We look to see if that file exists, and if so instantiate the class and call the relevant handler for the payment status.

-- hugh

 

[railer]

Thanks Hugh. Helpful to have the process explained.

Yes, in the second case, test mode was turned off. I made a live payment, but changed the amount to 0.01.

Yes, the form always sends the user to PayPal. We do have a calc element setup for the form if the user is editing the listing, which checks another payment history table to see if payment was successful the first time around, and if not, it redirects the user to PayPal to complete payment. But based on what you said, I don't think that would get in the way of the IPN.

As for the IPN handler code, we are using the default example code file installed with Fabrik. I don't think we have customized it. Do we need to? But again, the transaction completed in both my tests, so it must work as is.

 

[cheesegrits]

The default example code doesn't run, unless you rename it to from paypal_ipn_example.php to paypal_ipn.php ... and even then, all it would do would be on a "Pending" transaction, send an email to the payer_email and receiver_email given in the IPN callback args.

I was just checking that you hadn't renamed it and were running your own code on an IPN callback.

We do have a calc element setup for the form if the user is editing the listing, which checks another payment history table to see if payment was successful the first time around, and if not, it redirects the user to PayPal to complete payment

What IPN callback URL to you give for that redirect?

-- hugh

 

[railer]

Good to know about the example code file. We are not currently using it. We had the IPN PHP File set to the paypal_ipn_example.php option. Could that break the IPN notice? I've now sett the IPN PHP File parameter to "None Selected". Please advise if that is the correct setting.

My developer said to tell you "what we want is to save the basic information from the IPN in the table elements".

 

[cheesegrits]

No, you (probably) don't need the example. It's ... an example. The IPN handler stuff is optional, if you need to do more than just save the basic IPN info we provide the built in options for. The example shows how to send email to if the payment status is Pending.

So if all you need to do is save the info we have IPN element options for - txn-id, payment_amount, payment_status, address and subscr_id (if doing a subscription), you're good to go. If you need more than that, then you would need the custom IPN handler, and store that data yourself.

-- hugh