Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetpack JSON API access crashes PHP-FPM running PHP 7.1 #6629

Closed
michaelnordmeyer opened this issue Mar 10, 2017 · 55 comments
Closed

Jetpack JSON API access crashes PHP-FPM running PHP 7.1 #6629

michaelnordmeyer opened this issue Mar 10, 2017 · 55 comments
Labels
[Feature] WPCOM API [Pri] High [Type] Bug When a feature is broken and / or not performing as intended

Comments

@michaelnordmeyer
Copy link

Steps to reproduce the issue

  • Install Jetpack as the only plugin, enable JSON API, connect self-hosted blog to WordPress.com
  • Access self-hosted blog using either WordPress.com or the WordPress.com macOS app to display blog posts

What I expected

Blog posts are displayed.

What happened instead

App didn't display any content, only outlines of soon to be displayed blog posts.

Observation

PHP task crashes on every API access.

This behaviour also happens for other related API access.

Environment

  • WordPress 4.7.3
  • Jetpack 4.7
  • Ubuntu Server 16.10 64bit
  • nginx/1.11.9
  • PHP 7.1.2-4+deb.sury.org~yakkety+1
  • MySQL 5.7.17-0ubuntu0.16.10.1

Error logs

These are the nginx error logs intertwined with php7.1-fpm.log lines for better readability. Hostname was replaced by example.com.

2017/03/08 22:03:45 [error] 24445#24445: *122060 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.89.76, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D"

[08-Mar-2017 22:03:45] WARNING: [pool www] child 19742 exited on signal 11 (SIGSEGV) after 23.396837 seconds from start
[08-Mar-2017 22:03:45] NOTICE: [pool www] child 19746 started

2017/03/08 22:03:46 [error] 24445#24445: *122064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.110, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19719 exited on signal 11 (SIGSEGV) after 88.240395 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19747 started

2017/03/08 22:03:46 [error] 24445#24445: *122065 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.118, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19746 exited on signal 11 (SIGSEGV) after 1.566387 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19749 started

2017/03/08 22:03:46 [error] 24445#24445: *122069 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.138, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19747 exited on signal 11 (SIGSEGV) after 0.199491 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19751 started
@jeherve jeherve added [Feature] WPCOM API [Pri] High [Type] Bug When a feature is broken and / or not performing as intended labels Mar 10, 2017
@jeherve
Copy link
Member

jeherve commented Mar 10, 2017

cc @georgestephanis who's worked on other PHP 7.1 oddities lately.

@Ipstenu
Copy link

Ipstenu commented Mar 16, 2017

This is affecting me too BTW.

@georgestephanis
Copy link
Member

@Ipstenu @michaelnordmeyer :

  • Can you get any sort of a backtrace before the segfault?
  • Does it work if you disable opcache via init_set() ?
  • Does it work if you pull back to PHP 7.0?
  • Does it work if you disable the 'related posts' module?

@michaelnordmeyer
Copy link
Author

michaelnordmeyer commented Mar 20, 2017

@georgestephanis:

  • No, my PHP is not a debug build
  • Yes
  • It worked before switching from PHP 7.0 to 7.1 some weeks ago. If 7.0 would have been installed I would have added test info to the issue in the first place.
  • Only Jetpack with JSON API is enabled, no other Jetpack modules or 3rd-party plugins

@Ipstenu
Copy link

Ipstenu commented Mar 20, 2017

  • No :(
  • No
  • It works on PHP 7 just fine.
  • That happens to be disabled anyway

A quick and amusing check, on PHP 7.1 http://public-api.wordpress.com/rest/v1/sites/ipstenu.org shows up as a 404, but it works fine on PHP 7.

@georgestephanis
Copy link
Member

Heard back on an upstream bug report that I believe is related -- I'm going to try and track down what we can do to work around this, but for info for y'all in progress -- https://bugs.php.net/bug.php?id=74213

@ffPjrZUGXfcxuAj
Copy link

I'm also have this problem, switch to php 7.0.x and work

@michaelnordmeyer
Copy link
Author

After upgrading the environment today to

  • Jetpack 4.8.1
  • Ubuntu Server 17.04 64bit
  • PHP 7.1.4-1+deb.sury.org~zesty+1

there haven't been any crashes anymore. 👍

@ffPjrZUGXfcxuAj
Copy link

ffPjrZUGXfcxuAj commented Apr 18, 2017

Hi, i have this error again.
i think the problem is opcache

@jeherve jeherve reopened this Apr 19, 2017
@jeherve
Copy link
Member

jeherve commented Apr 19, 2017

@camaran Could you try to update PHP to 7.1.4? It includes the following fix, that may help you: https://bugs.php.net/bug.php?id=74213

Let me know how it goes!

@jeherve jeherve added the [Status] Needs Author Reply We would need you to make some changes or provide some more details about your PR. Thank you! label Apr 19, 2017
@ffPjrZUGXfcxuAj
Copy link

Hi, i installed php 7.1.4 but i still with the problem.
I can open a ticket on jetpack support if it's neccessary :-)

@rraptorr
Copy link

I am also experiencing the same crash with PHP 7.1.4 (from deb.sury.org), Wordpress 4.7.3 and Jetpack 4.8.2 running on Debian Jessie.

@jeherve
Copy link
Member

jeherve commented Apr 21, 2017

@camaran @rraptorr Do you get errors similar to the ones mentioned in the original report above?

@rraptorr
Copy link

I am getting segfaults on xmlrpc.php file. The problem is gone when switching back to php 7.0 or removing jetpack. This seems pretty similar to the original report.

I was testing every PHP 7.1 version, since PHP 7.1.0. I was always getting this segfault and I am getting it still with PHP 7.1.4.

@ChrisWiegman
Copy link
Contributor

After some testing I'm hitting the same issue:

Ubuntu 16.04
PHP v7.1.4-1+deb.sury.org~xenial+1
WordPress 4.7.4
Jetpack 4.8.2

The following in wp-config lets me perform a sync with Jetpack for the time being but I'm hoping this doesn't have to be permanent:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

@jeherve jeherve removed the [Status] Needs Author Reply We would need you to make some changes or provide some more details about your PR. Thank you! label Apr 23, 2017
@xyzulu
Copy link

xyzulu commented Apr 26, 2017

php 7.1.4 does not help with this issue
Before it get's closed again, perhaps let a few of us test ;)

@michaelnordmeyer
Copy link
Author

Looks like I have the problem again after a while of success with either the WordPress.com and iOS app. 😩

The only configuration change in the last couple of days was removing php-xdebug. The rest were ordinary OS updates and reboots because of an updated kernel.

Selectively turning off the opcache does work for me as well:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

@tanasebutcaru
Copy link

I have raised this issue last week.
If I switch back to php56 Wordpress Android App works, but with php7.1.4 doesn't work.

I have added opcache.enable = 0 to my domain's php.ini, but I cannot see any change.

@tanasebutcaru
Copy link

tanasebutcaru commented May 10, 2017

It is indeed a problem with opcache.
I have disabled opcache by creating a .user.ini file inside Wordpress's root dir containing the following line:
opcache.enable=0
and no more errors, Wordpress Android App works 💃

@AubreyHewes
Copy link

AubreyHewes commented May 15, 2017

Disabling opcache in wp-config.php for xmlrpc-only as @ChrisWiegman suggests fixes the problem;-) Cheers @ChrisWiegman !

The amount of headers sent/header validity still seems to me to be the possible root cause?

#3546
dokku/dokku#2607

My setup is a wp/jetpack via dokku (herokuish) ~ nginx-fpm

If you don't want the extra config in the wordpress config you can also blacklist xmlrpc in the opcache config...

http://php.net/manual/en/opcache.configuration.php#ini.opcache.blacklist-filename

#6106

@ffPjrZUGXfcxuAj
Copy link

i think now is fixed with php 7.1.5

@rraptorr
Copy link

Well, for me it segfaults just the same with 7.1.5

@xyzulu
Copy link

xyzulu commented Jun 9, 2017

@Ipstenu that isn't really a fix.. it's just a workaround IMHO

@Ipstenu
Copy link

Ipstenu commented Jun 9, 2017

I didn't call it a fix. I just said that it appears to be working as I'm one of the people facing this problem. If I don't report back with what has, and has not, worked, how can Automattic or anyone else know if their actual fix is headed in the right direction? :)

@xyzulu
Copy link

xyzulu commented Jun 9, 2017

@Ipstenu Sorry Mika, hope I didn't sound harsh.. this workaround has been mentioned before and will always work due to it disabling the feature in php that is not playing well with Jetpack in this instance. Going back months Automattic are well aware of this. (check the many threads where it is mentioned here on Github if you like)

Again.. no attack intended.. sorry for the wrong use of words

@Ipstenu
Copy link

Ipstenu commented Jun 9, 2017

Appears to always work.

More data, confirming the fix works in more environments, and replying to ones own bug reports, is considered good behavior in the open source community. I don't want to be DenverCoder9.

And it's spelled with an A. Not an E.

@xyzulu
Copy link

xyzulu commented Jun 9, 2017

It's an opcache error.. disabling opcache in php 7.1 will always work around this error

@Ipstenu
Copy link

Ipstenu commented Jun 9, 2017

If XMLRPC v OpCache is the only part that causes the core dump, sure.

@zinigor
Copy link
Member

zinigor commented Jun 9, 2017

Thank you all for the input! Just for the record, here's a bug report that I have submitted to PHP about this exact problem. It's not XMLRPC that causes the segfault, but apparently it's some code that gets called as part of the JSON API request that gets sent via XMLRPC.

If you happen to know why that code can cause a segfault with opcache, or maybe how I can narrow down the exact cause, I'll be very grateful:
https://bugs.php.net/bug.php?id=74543

@kevinlisota
Copy link

kevinlisota commented Jun 15, 2017

Has anyone tried this with PHP 7.1.6 yet? There is one opcache bug fix in that update.

Actually, I just tried 7.1.6 and the segfault bug still exists.

@lschuyler
Copy link

Also reported here 3261025-t

@zinigor
Copy link
Member

zinigor commented Jun 19, 2017

@kevinlisota just tested on 7.1.6 too, processes still segfault for me when responding to WordPress.com JSON API. What were the issues that were fixed with opcache? Can you post some links please?

@kevinlisota
Copy link

Here is the changelog for PHP 7.1.6: http://php.net/ChangeLog-7.php#7.1.6

There is one bugfix to opcache here: https://bugs.php.net/bug.php?id=74596

@HansVanEijsden
Copy link

Changelog PHP 7.1.7RC1: https://github.com/php/php-src/blob/PHP-7.1.7/NEWS

I don't know if it fixes it?

. Fixed bug #74663 (Segfault with opcache.memory_protect and
validate_timestamp). (Laruence)
. Revert opcache.enable_cli to default disabled. (Nikita)

@lschuyler
Copy link

Also reported in 3266547-t

@michakpl
Copy link

It's still not working in PHP 7.1.7

@kevinlisota
Copy link

@zinigor Any way to get PHP development team to look at the bug you filed? I see no activity on it

https://bugs.php.net/bug.php?id=74543

@kristarella
Copy link

Possible report 3332067-t

@zinigor
Copy link
Member

zinigor commented Jul 20, 2017

@kevinlisota A good way would be to create a simple way to reproduce the segfault, which I haven't been able to do. You can see how far I have been able to progress in the bug report, but I haven't had more time to dig in. If you are able to produce a contained reproduction case, that would help tremendously!

@lamdayap
Copy link

https://wordpress.org/support/topic/jetpack-causing-502-bad-gateway/

User is experiencing similar issues with PHP 7.0.15-0ubuntu0.16.04.4

@aheckler
Copy link
Member

614936-zen

Sent them this workaround: #6629 (comment)

@abcxyz-m
Copy link

630882-zen

Sent same work around as above. User would prefer to not troubleshoot.

@abcxyz-m
Copy link

635556-zen

Sent same work around as above.

@danjjohnson
Copy link

620933-zen sent suggested workarounds.

@abcxyz-m
Copy link

645154-zen

Sent same work around as above.

@mrailton
Copy link

Currently running on PHP 7.1.8 and having same issue. Disabling opcache works as a workaround

@dausruddin
Copy link

PHP 7.1.9 and having the same problem.
Also disabling opcache fixed it too.

@liuyigh
Copy link

liuyigh commented Sep 5, 2017

PHP 7.1.8 on serverpilot and having the same problem. Trouble shooting for hours until reading this thread. Switch to 7.0 on serverpilot, then it works right away. Thanks, guys!

@liuyigh
Copy link

liuyigh commented Sep 5, 2017

The Jetpack people noticed me about the freshly baked 5.3 update today:

https://jetpack.com/2017/09/05/jetpack-5-3-php-7-1-compatibility/

Now works well with php 7.1. Hooray!

@jeherve jeherve closed this as completed Sep 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Feature] WPCOM API [Pri] High [Type] Bug When a feature is broken and / or not performing as intended
Projects
None yet
Development

No branches or pull requests