Table and Query Classes good or unnecessary?

hey martin.

we agree.. although I did have to go try to research if postgres 10 and their parallel workers were thread or processes. We have postgres 10 running and I”m not too sure which way they went. I finally found an article mentioning Roberta Haas (a postgres contributor) and it seems to imply threads for this feature.

Language workers yes… .. thats the next step-forward request.. I’d really like omnis to run omnis – what a time saver that would be for a number of things . I think a lot of people would get something out of that approach. The question has been asked of engineering.

I can’t belive how much we use the worker pattern in things these days. its become so natural.

Doug Easterbrook
Arts Management Systems Ltd.
mailto:doug@artsman.com
www.artsman.com
Phone (403) 650-1978

see you at the third annual users conference
tickets.proctors.org/TheatreManager/95/online?performance=29086 <tickets.proctors.org/TheatreManager/95/online?performance=29086>

> On Mar 20, 2018, at 4:16 PM, Marten Verhoeven <m.verhoeven@van-beek.nl> wrote:
>
> Hi Doug,
>
> Actually, Postgres is starting to support multi core query execution (so one query using multiple cores) 🙂
>
> www.postgresql.org/docs/10/static/parallel-query.html
>
> I do agree though that Omnis should not focus on language level multi threading (so no async/await-like support). But fully separate omnis execution stacks (with separate CRB etc.) should be the (first) focus. Not able to create UI, but just handle a different task, called as a worker, or like a remote task, but pre-initialized (and not closed when the call is completed).
>
> I love the sql worker objects (use them a lot) and will use the other workers as soon as I can upgrade to 8.1. If I can run Omnis Code as a worker I would definitely use that (a lot). So, yes please:-)
>
> What Omnis also could do is use multi-threading to enhance some internal processing. Searching (or maybe even manipulating) large lists could benefit from multi-threading for example.
>
> With kind regards,
>
> Marten Verhoeven
>
> —–Oorspronkelijk bericht—–
> Van: omnisdev-en [mailto:omnisdev-en-bounces@lists.omnis-dev.com] Namens Doug Easterbrook
> Verzonden: dinsdag 20 maart 2018 16:28
> Aan: OmnisDev List – English <omnisdev-en@lists.omnis-dev.com>
> Onderwerp: Re: Table and Query Classes good or unnecessary?
>
> hi Stene.
>
> I looked at the graphic.. and precisely my point.
>
>
> you don’t want omnis taking up more that one core. A Postgres process does not take up more than one core. Python does not take up more than one core.
>
>
> postgres has many processes that take up multiple cores. Similarly, Omnis can have multiple processes that take up more than one core.
>
>
> writing applications that follow the multiple process paradigm is far safer, far easier, more isolated, more stable than multi-threaded applications. It can be done quicker since you don’t worry about cross talk in threads or synchronization issues.
>
>
> Omnis (no offence to the engineering folks) would need a lot of overhead to make it multi threaded. They tried with async methods — but it was far to much to handle in a graceful way. We would, as programers, have far far more benefit if the focus was performance within omnis… and let people do multi processing for handing off things like omnis job workers.
>
>
>
> A case for OMNIS LANGUAGE WORKER
>
> in fact, the think I think we’d like most in omnis these days (which plays into the middle tier stuff) is OMNIS LANGUAGE WORKERS. You aren’t really saying it, but I think thats what you are wanting or what will get you to the end goal of using more cores in any processor.
>
>
> in other words, we’d like a worker class that we can invoke from omnis to tell something to go and run a worker in any language.
>
> That worker could be Omnis (eg — go run an omnis report, or go and run some code I’ve written in omnis — but start up another omnis do do it). That is multi process, but darn close to multi threaded. except, I’m clearly saying its multi process for the benefit it provides
>
> That worker could be python (the PDF printer in studio 8 is just such as beast — it tells python to print a PDF)… so why not have a language worker do things like made a 2D QR code and return an image. That steps into the open source world nicely …
>
> A worker could be a C++ program (eg convert my image to a tiff)
>
>
>
>
>
>
> Omnis is a great GUI tool … and using it to integrate open source snippets from any language (including omnis) is only of benefit.
>
>
> I have to restate, I don’t want omnis to become inherently multi-threaded. Thats a bad bad move. and yes, I know it doesn’t have to take advantage of more than one core. We need tools to let us do things on other cores.
>
> those tools are:
> – worker objects (in the dams and web components)
> – language workers, where we can hand a task off to a process, short lived as it may be, in any language, including omnis
>
> that will get us developers taking advantage of the cpu’s in front of us.
>
>
> and so, I will respectfully disagree with the assertion that we cannot take advantage of a multi core machine. I believe we are.. since I can run stuff in omnis and out setup to pin cpu’s to almost all of what is available.
>
>
> I’m going to watch this kind of setup happen tomorrow … we have one venue that is doing an onsale for tickets that should last about 15 minutes.. they will have something like 75 omnis processes, 32 python processes, 300 database processes, 2 nginx processes on multiple machines and I suspect we’ll see very high utilization of multiple machines.
>
> again, not because of multi threaded —but because we are making effective use of a range of tools that are on the multi-process bandwagon.
>
>
>
>
>
>
> Doug Easterbrook
> Arts Management Systems Ltd.
> mailto:doug@artsman.com
> www.artsman.com
> Phone (403) 650-1978
>
>
>
>
> see you at the third annual users conference
> tickets.proctors.org/TheatreManager/95/online?performance=29086 <tickets.proctors.org/TheatreManager/95/online?performance=29086>
>
>> On Mar 20, 2018, at 8:06 AM, Sten-Erik Björling <s-e.bjorling@enviro.se> wrote:
>>
>> Hi Doug,
>>
>> The problem here is that my graphic did not get through. Look at the url below:
>>
>> envirodata.se/docs/Omnis/Omnis_processor_load.png
>> <envirodata.se/docs/Omnis/Omnis_processor_load.png>
>>
>> The point here is that Omnis do not allow for more than one core computing capacity even if there are several more cores available and that the OS demonstratively can handle the load delegation between several cores. What would happen if Omnis got this limitation taken out?
>>
>> I am fully aware of the approaches of using multiple instances of omnis running both with Omnis own load balancing and your approach and I am also fully aware of the potential coming from worker objects. That is not the problem that I am aiming at. Getting Omnis to more fully utilise the existing cores will increase the performance even more when using worker objects – kindly note the low utilization of ONE postgres connection in this case for this test – and this test where not using worker objects.
>>
>> Today when very powerful multicore computers can be bought for relatively little money it is a wast that this computing capacity cannot be fully utilised by Omnis.
>>
>> Take care, all the best…
>>
>> Stene
>>
>> ______
>>
>>> 20 mars 2018 kl. 14:33 skrev Doug Easterbrook <doug@artsman.com>:
>>>
>>> hi Stene:
>>>
>>> careful on this — its a bit misleading since there is an apples an
>>> oranges comparison of MULTI THREADED vs MULTI PROCESS
>>>
>>> You are right that ONE omnis cannot consume more than 100% of a
>>> single CPU core because it is SINGLE THREADED
>>>
>>> yet, we demonstrated ONE omnis using 350% of a 4 core machine (call that 400%) — using the pgsql dam workers about 7 or 8 years ago at euromnis with an early demo of threaded workers. Meaning a SINGLE THREADED omnis with MULTI THREADED xcomps could handle some serious data acquisition using an async model. Thats been extended to the web workers.
>>>
>>>
>>>
>>> What Martin was saying to Reg it a truism of MULTI_PROCESSING. if you sparf up enough SINGLE THREADED processes, you can also consume a lot of the CPU. In tests, we’ve been able to spark up 16 or more omnis processes and really max out a quad core machine — in that scenario, omnis, as multi-process, could do a lot of web requests. (we’ve also sparked up more on larger machines to try to find limits).
>>>
>>>
>>>
>>> so, the commentary on the slide, which says:
>>>
>>>> The processing load for Omnis is distributed amongst 4 different cores but is not allowed to consume more than 100% of processing power – but the max available processing power is between 400 – 800% (depending on utilisation of hyper threads).
>>>
>>>
>>>
>>> can be altered to be this…
>>>
>>> ONE Omnis, can use 100% of ONLY one CPU for omnis 4GL CODE because it is SINGLE THREADED.
>>>
>>> ONE Omnis, with postgres or web workers, can be made to use all of
>>> power on ALL CPU’s since — omnis will stay in the one CPU — the
>>> workers will use the other cpu’s because they are MULTI PROCESS
>>>
>>>
>>> MANY omnis, running in a multi-process environment can use ALL of the CPU’s in the machine, quite handily.
>>>
>>>
>>> Same as any other language, the developer is left to decide how to mix MULTI PROCESSING and MULTI THREADING to achieve what they need.
>>>
>>>
>>>
>>> you are dead in the water, with any language, if you stick to a single processing model since you are avoiding scaleability of the other cpus.
>>>
>>>
>>> The above is how we wrote our web services (multi-Process, multi threaded) in omnis.
>>>
>>>
>>> We had to use a similar development model in Python to get multi process working for web services – in other words, we spark up a lot of python processes to listen to NGINX web requests. That uses all the CPU cores – because of multi processing. The exact same thing was done with omnis in the past where multiple omnis listened to an apache server – and that also worked because of multi processing.
>>>
>>> similar design. omnis took 400% of the CPU.
>>>
>>>
>>>
>>> I’m neither defending omnis nor promoting python.
>>>
>>>
>>> what I am suggesting is that omnis as a server process:
>>> — which is lightweight to start (headless linux — looking forward to
>>> this) and — and continues to get some speed optimizations from
>>> engineering (it is getting faster I see) and — has a front end that
>>> distributes load (eg NGINX or Apache)
>>>
>>> could function similarly to python.
>>>
>>>
>>> Omnis will never be a capable MULTI-THREADED language. I hope that nobody even tries. The power, as Martin says, is using these tools in mutli-process design.
>>>
>>>
>>> not sure if that makes sense.
>>>
>>>
>>> It took us a few years to get over the ‘make one omnis do all the work’ instead of make ‘many omnis share all the work’. we were boxed in a paradigm and it cost us a few years of development effort …. once we broke out .. its been a breath of fresh air. so, as an example, we are going to implement multi-prcess omnis report servers .. and we can fire a lot of reports to a server and get them done in the background, simultaneously, multi-process style.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Doug Easterbrook
>>> Arts Management Systems Ltd.
>>> mailto:doug@artsman.com
>>> www.artsman.com
>>> Phone (403) 650-1978
>>
>> Sten-Erik Björling
>> Enviro Data
>> Kyrkogatan 5A 2 tr
>> SE-972 32 Luleå
>> Sweden
>>
>> E-Mail: s-e.bjorling@enviro.se <mailto:s-e.bjorling@enviro.se>
>> Mobile: +46-70-655 11 72
>> Wire: @stenerikbjorling
>> Skype: stenerikbjorling
>> iChat: stene@mac.com
>> FaceTime: stene@mac.com <mailto:stene@mac.com> Hotmail / Messenger:
>> stenerikbjorling@hotmail.com <mailto:stenerikbjorling@hotmail.com>
>> GMail: stenerikbjorling@gmail.com <mailto:stenerikbjorling@gmail.com>
>>
>> ______
>>
>> This email and any files transmitted with it are confidential, may be legally privileged and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, please note that any use, distribution, or reproduction of the contents of this email is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by return email and destroy all copies of the original message including any attachments thereto. Thank you.
>>
>> Please note that we take reasonable precautions to prevent the transmission of viruses; however, we cannot guarantee that this email or its attachments are free from viruses. We only send and receive emails on the basis that we are not be liable for any loss or damage resulting from the opening of this message and/or attachments.
>>
>> _____________________________________________________________
>> Manage your list subscriptions at lists.omnis-dev.com
>> <lists.omnis-dev.com/> Start a new message -> mailto:omnisdev-en@lists.omnis-dev.com <mailto:omnisdev-en@lists.omnis-dev.com>
> _____________________________________________________________
> Manage your list subscriptions at lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en@lists.omnis-dev.com

_____________________________________________________________
Manage your list subscriptions at lists.omnis-dev.com
Start a new message -> mailto:omnisdev-en@lists.omnis-dev.com