Weird repeat group behavior with databasejoin

[kouwearie]

Theres definitly a bug. It is caused by some variable names which occur as a function parameters and is then redeclared using a var assignment.
Probably a buttonscript

 

[cheesegrits]

I'm not gettng why you paste a quote from me from another thread, about another issue (a minor ordering bug in a work-in-progress function), into this thread about an, in my opinion, critical bug in the system which ends up deleting data which users do not want deleted.

Or rather, I can think of a reason why, but am having a hard time believing that what I'm thinking of is actually true

I'm simply pointing out, in a roundabout way, that Fabrik is entirely funded by subscriptions. You mentioned client sites, so I presume you are using Fabrik as one of the tools you use to make a living. You aren't prepared to help fund fixing minor issues, so I'm hoping you might be prepared to help fund fixing a major issue, so you can continue to use the tool, and I can pay my rent. Win win.

-- hugh

 

[cheesegrits]

BTW, I am working on this as time permits, it's just a pig to debug, as it seems to be a "Heisenbug", in that the behavior changes when observed, like putting a breakpoint in the code.

It's just like any Fabrik issue. The more people who support development through subscriptions, the more time I can spend working on stuff like this, rather than doing billable custom coding to make up the difference.

-- hugh

 

[cheesegrits]

OK, I just spent another hour working on this, trying to get my test groups to fail. Out of about 100 attempts, I managed to get one fail, but didn't have a breakpoint set anywhere useful.

And that's where I'm kind of stuck. I need to find a way of reliably replicating this issue, so i can actually debug it.

-- hugh

 

[cheesegrits]

*sigh*

Of course, right after I posted that, the first time I tried it again, it failed. But now I can't get it to fail again. I'm going to have to walk away from this for a while before I throw my laptop out the window.

-- hugh

 

[PvN94]

I'm certainly willing to help fund you, Fabrik is a great tool and even with only the free support you guys do a good job. It's just that with the copying of the quote it came off as a very passive agressive way of saying I'll have to pay up for you to keep working on this, which is exactly not what I'm used to here.

As for the issue itself, it used to reliably fail when I had a form with at least 3 rows in the repeat group, removing any row and saving would empty another row. But in the latest version I have installed, about a week old from github, that seems to occur way less. I've clicked around a bit and for me when I add 2 rows, and then remove those 2 rows and save, I'm getting issues most of the time. Sometimes just a dbjoin element that's emptied, and sometimes an entire row that's gone.

 

[cheesegrits]

I'm certainly willing to help fund you, Fabrik is a great tool and even with only the free support you guys do a good job. It's just that with the copying of the quote it came off as a very passive aggressive way of saying I'll have to pay up for you to keep working on this,

It was slightly passive aggressive, as I was somewhat confused by the comment about it not being worth a sub to fix little things, while you had a thread open asking for a big thing to be fixed. If it's not worth people having a sub to pay for my time to fix little things or big things, I'm not entirely sure how they think Fabrik is supposed to survive.

which is exactly not what I'm used to here.

Well, that's part of the problem. Despite my occasional passive aggression, I'm basically a nice guy, I like helping people, and I've spent far too long giving away help for free (I think there's something like 20,000 posts from me in Community over the years), but I'm at the point where I just can't do that any more. Less and less people are paying for subs, it's not paying the bills. Partly my fault as I've spent 10 years building the expectation that'll I'll just fix anything and everything for free in Community, so there's little incentive for people to pay for a sub. And I just have to quit doing that. It's open source software, people can use it as-is, if there's issues they need fixed, they'll need to be actively contributing to development and support - either through debugging and coding, and/or financially.

This isn't aimed at you specifically, I'm just trying to reset expectations for everyone. I have a couple of PM threads going at the moment with people in Community, who use Fabrik to do commercial work for clients, who are berating me for not fixing bugs to their deadlines. "It's a bug, you should fix it, I shouldn't have to pay for that". And I'm just blown away, I don't understand that attitude, and it's kind of upsetting me, so I'm starting to dig my heels in a little.

I'll continue trying to debug this one, but I've already spent about 6 hours on it, and I'm having to concentrate on work which directly helps me pay my bills atm.

-- hugh

 

[PvN94]

I do think there is a bit of a grey area when talking about bugs in the basic funtionality. Because even though I can use Fabrik for free, and I know and understand that you need the money from subscriptions to pay your bills, the first reaction is still that the basic funtionality should 'just work'.

That being said, you're totally right that you have to pay the bills, and have to give priority to work that pays them, so I've taken out a support sub in the hopes that you can give more priority to this issue.


About the issue, I've seen in a form with 4 rows in my repeat group that deleting the first row gives no problems, but deleting the last row does, and it does so every time. Don't know why that would be, but maybe it helps you out.

 

[cheesegrits]

I do think there is a bit of a grey area when talking about bugs in the basic funtionality. Because even though I can use Fabrik for free, and I know and understand that you need the money from subscriptions to pay your bills, the first reaction is still that the basic funtionality should 'just work'.

If Fabrik was a commercial package which you were paying for, that would be a reasonable expectation. But it's not. It's open source, provided "as is", funded entirely by those who find it useful.

Ultimately, Fabrik is a toolbox, which saves a huge amount of time in constructing applications. But it will always have bugs. It's about a third of a million lines of code, which has grown organically over a decade, with (at last count) about 140 plugins that can interact in several trillion permutations. So it's always going to be a collaborative effort between us and people who use it to make their living. And it's up to them / you how much it is worth to you as a toolbox.

I haven't looked at the app(s) you are building, but Fabrik can save hundreds of hours coding on even a relatively simple app. And typically when you run into an issue like this, it's usually something you would run into when trying to roll your own code. Adding / removing repeated groups of varied form inputs and associated DOM structures on the fly from first principles, which in turn updates relational database tables, is not a trivial thing to do. So if you have hit a mission critical problem with this, your choice is either to not use Fabrik and implement it all yourself, or for us to track and fix it for you. How quickly we track and fix it, as discussed, depends in part on what you contribute to that effort.

About the issue, I've seen in a form with 4 rows in my repeat group that deleting the first row gives no problems, but deleting the last row does, and it does so every time. Don't know why that would be, but maybe it helps you out.

Yes, if you have a reliably repeatable scenario, that would be very useful. In the hours of work I've put into this, with hundreds of tests, I've managed to tickle this bug about 3 times.

What I suspect is there is an asynchronous issue happening. All this code is event driven, and JS events are asynchronous. Which can lead to problems if there is some code which relies on other code having run to completion. Which is often to do with the "trillions of permutations" mentioned above. Certain element types or combinations thereof may do stuff when being duplicated that wind up throwing curve balls.

Can you point me at that form. Also enable Debug JS in the Fabrik global settings, so I can debug the uncompressed JS.

-- hugh

 

[cheesegrits]

Don't forget to give me those details so I can work on this.

-- hugh

 

[PvN94]

I've set up a separate website for you on another server, with a fresh Joomla 3.6.4 install and a fresh Fabrik install fully updated from github. In 'my sites' i've put a link to a simple form set up with a repeat group, and in that group one dbjoin and 2 fields. I've added the login details for a Super User as well. Feel free to log in to the back-end to look over and change some settings, it's completely empty except for this test data, and all settings are on default.

What I do is make sure that there is 4 rows in the repeat group, with a value selected in the dbjoin and some values filled in in the fields that help me see which value the join should have in that row. When that data is saved I open the record, add 1 or 2 rows in the repeat group, then immediately delete those rows I added, and click apply. About half the time, after saving, the first row in the repeat group will be empty.

It used to be that I could get multiple empty rows and even some cases where dbjoin values where switched (when I select e.g. 'Join4' in the dbjoin I put something like '4.1' and '4.2' in the fields, and I would get results where the join said 'Join1' and the fields '4.1' and '4.2'), but I can't get that to happen in the latest git version, so there seems to have been a change that at least fixed some of the issue.

 

[cheesegrits]

OK, thanks, I appreciate your effort and time.

I'll try and find time soon to work on this.

Just out of interest, does it make any difference how quickly you add and delete the groups? By which I mean the interval between hitting the buttons. I recall a while back we worked on a similar issue, which turned out to be asyncronous issues when hitting those buttons rapidly. The asyncronous nature of JS meant that the first add/remove hadn't completed before the second one ran, so the group numbering (element ids, etc) got messed up. I added some "gating" to disable the buttons after pressing one until the operation completes, so in theory that shouldn't happen any more. But I was never entirely happy with that code.

So try doing it slowly, several seconds between clicks, and see if that affects behavior.

-- hugh

 

[PvN94]

I can't seem to find any difference between slow or fast clicks, or even waiting up to a minute between clicks. It's also not only when adding and deleting that it happens, sometimes it's also when just deleting the last of four rows, without adding rows first. This seems to be less common, but I have seen it happen.

It seems to me like there is indeed something wrong with the group numbering and that it's confusing the newly added group for the first group. When you look at the html element id's. Like if you add a 5th row all the element id's should be like 'table_name___element_name_4', but the fabrikElementContainer div gets a class with 'table_name___element_name_0'.


I just took a look at the table for the repeat group before and after the error, and the logs (from the logs form plugin) and here's what I found:

This is the table with 4 good rows of data:

id | parent_id | dbjoin | f1  | f2
47 | 1         | 2      | 2.1 | 2.2
51 | 1         | 1      | 1.1 | 1.2
54 | 1         | 3      | 3.1 | 3.2
56 | 1         | 4      | 4.1 | 4.2

And then after I added and removed a row, and found an empty row in the form, the table looks like this:

id | parent_id | dbjoin    | f1  | f2
47 | 1         | 2         | 2.1 | 2.2
51 | 1         | 1         | 1.1 | 1.2
54 | 1         | ["4","3"] | 3.1 | 3.2
57 | 1         | NULL      |     |

So somehow rows 54 and 56 are "merged", and we get a new, empty, row 57. The row 54 now displays the dbjoin as being 4, even though it was originally 3.

The data comparison from the logs plugin says this:

COMPARE_DATA_CHANGE_ON id <br/>
COMPARE_DATA_FROM 56 <br/>
COMPARE_DATA_TO 47 <br/>

COMPARE_DATA_CHANGE_ON DbJoin <br/>
COMPARE_DATA_FROM Join4 <br/>
COMPARE_DATA_TO Join2 <br/>

COMPARE_DATA_CHANGE_ON F1 <br/>
COMPARE_DATA_FROM 4.1 <br/>
COMPARE_DATA_TO 2.1 <br/>

COMPARE_DATA_CHANGE_ON F2 <br/>
COMPARE_DATA_FROM 4.2 <br/>
COMPARE_DATA_TO 2.2 <br/>

So that seems to edit row 56 to be the same as 47, including the id, which should give a duplicate row 47, but I'm guessing that isn't the case because the database rejects the id value 47, as that would already exist.
The logs table is filled with logs like this one, with rows being edited to the values of other rows, with sometimes all the "to" values being empty except for the id.

 

[PvN94]

Have you had some time to look at this already?